2024-03-29T10:23:52.942 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T10:23:52.943 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --verbose --owner scheduled_amathuri@teuthology --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622641 -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622641/orig.config.yaml 2024-03-29T10:23:52.979 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:23:53.073 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622641 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622641' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi057.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOIvQcBjWXDczcrEinUPtbVNMsWv8poI/HVsMsTY3lZhdJ/G2+kuaCcn8AlPIGoyBeouP2tDxUXV2DDKLqHqJMU= smithi179.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOAEcOUl39u6xHmX9AOi+6H5hugRXuFb1wwZYRXKWzwhnLEYf6371TH0sVoDdYEUpoblQup/LyAf0MeJ87SVye4= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T10:23:53.073 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T10:23:53.074 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T10:23:53.074 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T10:23:53.075 INFO:teuthology.task.internal:Checking packages... 2024-03-29T10:23:53.099 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T10:23:53.100 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T10:23:53.100 INFO:teuthology.packaging:ref: None 2024-03-29T10:23:53.100 INFO:teuthology.packaging:tag: None 2024-03-29T10:23:53.100 INFO:teuthology.packaging:branch: main 2024-03-29T10:23:53.100 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:23:53.100 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T10:23:53.375 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T10:23:53.376 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T10:23:53.384 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T10:23:53.384 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T10:23:53.430 INFO:teuthology.task.internal:Saving configuration 2024-03-29T10:23:53.444 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T10:23:53.458 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:23:53.484 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622641', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:18:40.038645', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOIvQcBjWXDczcrEinUPtbVNMsWv8poI/HVsMsTY3lZhdJ/G2+kuaCcn8AlPIGoyBeouP2tDxUXV2DDKLqHqJMU='} 2024-03-29T10:23:53.508 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622641', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:18:40.037293', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOAEcOUl39u6xHmX9AOi+6H5hugRXuFb1wwZYRXKWzwhnLEYf6371TH0sVoDdYEUpoblQup/LyAf0MeJ87SVye4='} 2024-03-29T10:23:53.509 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T10:23:53.515 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T10:23:53.515 INFO:teuthology.task.internal:roles: ubuntu@smithi179.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T10:23:53.515 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T10:23:53.608 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f843831a0d0>, signals=[15]) 2024-03-29T10:23:53.608 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T10:23:53.614 INFO:teuthology.task.internal:Opening connections... 2024-03-29T10:23:53.614 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi057.front.sepia.ceph.com 2024-03-29T10:23:53.616 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:23:53.728 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi179.front.sepia.ceph.com 2024-03-29T10:23:53.729 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:23:53.819 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T10:23:53.826 DEBUG:teuthology.orchestra.run.smithi057:> uname -m 2024-03-29T10:23:53.844 INFO:teuthology.orchestra.run.smithi057.stdout:x86_64 2024-03-29T10:23:53.844 DEBUG:teuthology.orchestra.run.smithi057:> cat /etc/os-release 2024-03-29T10:23:53.902 INFO:teuthology.orchestra.run.smithi057.stdout:NAME="CentOS Stream" 2024-03-29T10:23:53.902 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION="9" 2024-03-29T10:23:53.902 INFO:teuthology.orchestra.run.smithi057.stdout:ID="centos" 2024-03-29T10:23:53.902 INFO:teuthology.orchestra.run.smithi057.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:23:53.902 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_ID="9" 2024-03-29T10:23:53.903 INFO:teuthology.orchestra.run.smithi057.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:23:53.903 INFO:teuthology.orchestra.run.smithi057.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:23:53.903 INFO:teuthology.orchestra.run.smithi057.stdout:ANSI_COLOR="0;31" 2024-03-29T10:23:53.903 INFO:teuthology.orchestra.run.smithi057.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:23:53.903 INFO:teuthology.orchestra.run.smithi057.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:23:53.903 INFO:teuthology.orchestra.run.smithi057.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:23:53.903 INFO:teuthology.orchestra.run.smithi057.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:23:53.903 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:23:53.903 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:23:53.904 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server 2024-03-29T10:23:53.932 DEBUG:teuthology.orchestra.run.smithi179:> uname -m 2024-03-29T10:23:53.950 INFO:teuthology.orchestra.run.smithi179.stdout:x86_64 2024-03-29T10:23:53.950 DEBUG:teuthology.orchestra.run.smithi179:> cat /etc/os-release 2024-03-29T10:23:54.007 INFO:teuthology.orchestra.run.smithi179.stdout:NAME="CentOS Stream" 2024-03-29T10:23:54.007 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION="9" 2024-03-29T10:23:54.007 INFO:teuthology.orchestra.run.smithi179.stdout:ID="centos" 2024-03-29T10:23:54.007 INFO:teuthology.orchestra.run.smithi179.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:23:54.007 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION_ID="9" 2024-03-29T10:23:54.007 INFO:teuthology.orchestra.run.smithi179.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:23:54.007 INFO:teuthology.orchestra.run.smithi179.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:23:54.008 INFO:teuthology.orchestra.run.smithi179.stdout:ANSI_COLOR="0;31" 2024-03-29T10:23:54.008 INFO:teuthology.orchestra.run.smithi179.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:23:54.008 INFO:teuthology.orchestra.run.smithi179.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:23:54.008 INFO:teuthology.orchestra.run.smithi179.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:23:54.008 INFO:teuthology.orchestra.run.smithi179.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:23:54.008 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:23:54.008 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:23:54.008 INFO:teuthology.lock.ops:Updating smithi179.front.sepia.ceph.com on lock server 2024-03-29T10:23:54.032 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T10:23:54.041 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T10:23:54.092 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T10:23:54.092 DEBUG:teuthology.orchestra.run.smithi057:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:23:54.095 DEBUG:teuthology.orchestra.run.smithi179:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:23:54.111 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T10:23:54.134 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T10:23:54.134 DEBUG:teuthology.orchestra.run.smithi057:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:23:54.152 DEBUG:teuthology.orchestra.run.smithi179:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:23:54.246 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T10:23:54.302 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T10:23:54.318 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T10:23:54.318 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T10:23:54.318 DEBUG:teuthology.orchestra.run.smithi057:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:23:54.319 DEBUG:teuthology.orchestra.run.smithi179:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:23:54.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:23:54.336 DEBUG:teuthology.orchestra.run.smithi179:> uname -r 2024-03-29T10:23:54.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:23:54.337 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2024-03-29T10:23:54.393 INFO:teuthology.orchestra.run.smithi179.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:23:54.393 INFO:teuthology.task.kernel:Running kernel on smithi179: 5.14.0-361.el9.x86_64 2024-03-29T10:23:54.393 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum install -y kernel 2024-03-29T10:23:54.395 INFO:teuthology.orchestra.run.smithi057.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:23:54.395 INFO:teuthology.task.kernel:Running kernel on smithi057: 5.14.0-361.el9.x86_64 2024-03-29T10:23:54.395 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum install -y kernel 2024-03-29T10:23:56.564 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T10:23:56.599 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-29T10:23:58.315 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-03-29T10:23:58.575 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-9 - BaseOS 2.3 MB/s | 2.0 MB 00:00 2024-03-29T10:23:59.299 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - BaseOS 21 MB/s | 8.0 MB 00:00 2024-03-29T10:24:01.452 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-03-29T10:24:05.334 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - CRB 17 MB/s | 6.1 MB 00:00 2024-03-29T10:24:06.910 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-03-29T10:24:08.171 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - BaseOS 866 kB/s | 8.0 MB 00:09 2024-03-29T10:24:08.473 INFO:teuthology.orchestra.run.smithi179.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-29T10:24:10.658 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-29T10:24:13.462 INFO:teuthology.orchestra.run.smithi179.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-29T10:24:15.752 INFO:teuthology.orchestra.run.smithi179.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:24:15.753 INFO:teuthology.orchestra.run.smithi179.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:24:15.846 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:24:15.850 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:24:15.851 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-29T10:24:15.851 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:24:15.851 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T10:24:15.851 INFO:teuthology.orchestra.run.smithi179.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:24:15.851 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-29T10:24:15.851 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:24:15.851 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:24:15.851 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:24:15.851 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:24:15.851 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T10:24:15.851 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:24:15.851 INFO:teuthology.orchestra.run.smithi179.stdout:Install 4 Packages 2024-03-29T10:24:15.851 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:24:15.852 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 76 M 2024-03-29T10:24:15.852 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 123 M 2024-03-29T10:24:15.852 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T10:24:16.005 INFO:teuthology.orchestra.run.smithi179.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T10:24:16.855 INFO:teuthology.orchestra.run.smithi179.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 15 MB/s | 15 MB 00:01 2024-03-29T10:24:17.539 INFO:teuthology.orchestra.run.smithi179.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 20 MB/s | 34 MB 00:01 2024-03-29T10:24:18.072 INFO:teuthology.orchestra.run.smithi179.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 13 MB/s | 28 MB 00:02 2024-03-29T10:24:18.075 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:24:18.076 INFO:teuthology.orchestra.run.smithi179.stdout:Total 34 MB/s | 76 MB 00:02 2024-03-29T10:24:18.076 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T10:24:18.454 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T10:24:18.454 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T10:24:19.286 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T10:24:19.286 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T10:24:21.608 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T10:24:22.201 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:24:22.286 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:24:22.337 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - CRB 787 kB/s | 6.1 MB 00:07 2024-03-29T10:24:24.205 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 16 kB 00:00 2024-03-29T10:24:25.032 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:24:25.185 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:24:29.889 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:24:30.143 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:24:34.881 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:25:04.561 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:25:04.562 INFO:teuthology.orchestra.run.smithi179.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:25:04.562 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:25:04.604 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:25:04.948 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:25:04.948 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:25:04.948 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:25:04.948 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:25:05.337 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:25:05.337 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:25:05.337 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T10:25:05.337 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:25:05.337 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:25:05.338 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:25:05.338 INFO:teuthology.orchestra.run.smithi179.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:25:05.338 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:25:05.338 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:25:05.674 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:25:05.769 INFO:teuthology.orchestra.run.smithi179.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:25:05.770 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:25:05.770 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:25:05.770 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:25:05.770 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:25:05.771 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T10:25:05.771 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi179.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:25:05.771 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:25:05.851 INFO:teuthology.orchestra.run.smithi179.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:25:05.851 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:25:05.851 DEBUG:teuthology.orchestra.run.smithi179:> sudo rpm -qi grub2-tools 2024-03-29T10:25:05.892 INFO:teuthology.orchestra.run.smithi179.stdout:Name : grub2-tools 2024-03-29T10:25:05.892 INFO:teuthology.orchestra.run.smithi179.stdout:Epoch : 1 2024-03-29T10:25:05.892 INFO:teuthology.orchestra.run.smithi179.stdout:Version : 2.06 2024-03-29T10:25:05.892 INFO:teuthology.orchestra.run.smithi179.stdout:Release : 68.el9 2024-03-29T10:25:05.892 INFO:teuthology.orchestra.run.smithi179.stdout:Architecture: x86_64 2024-03-29T10:25:05.892 INFO:teuthology.orchestra.run.smithi179.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:25:05.892 INFO:teuthology.orchestra.run.smithi179.stdout:Group : Unspecified 2024-03-29T10:25:05.892 INFO:teuthology.orchestra.run.smithi179.stdout:Size : 8264254 2024-03-29T10:25:05.892 INFO:teuthology.orchestra.run.smithi179.stdout:License : GPLv3+ 2024-03-29T10:25:05.892 INFO:teuthology.orchestra.run.smithi179.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:25:05.892 INFO:teuthology.orchestra.run.smithi179.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:25:05.892 INFO:teuthology.orchestra.run.smithi179.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:25:05.892 INFO:teuthology.orchestra.run.smithi179.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:25:05.893 INFO:teuthology.orchestra.run.smithi179.stdout:Packager : builder@centos.org 2024-03-29T10:25:05.893 INFO:teuthology.orchestra.run.smithi179.stdout:Vendor : CentOS 2024-03-29T10:25:05.893 INFO:teuthology.orchestra.run.smithi179.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:25:05.893 INFO:teuthology.orchestra.run.smithi179.stdout:Summary : Support tools for GRUB. 2024-03-29T10:25:05.893 INFO:teuthology.orchestra.run.smithi179.stdout:Description : 2024-03-29T10:25:05.893 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:25:05.893 INFO:teuthology.orchestra.run.smithi179.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:25:05.893 INFO:teuthology.orchestra.run.smithi179.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:25:05.893 INFO:teuthology.orchestra.run.smithi179.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:25:05.893 INFO:teuthology.orchestra.run.smithi179.stdout:hardware devices. 2024-03-29T10:25:05.893 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:25:05.893 INFO:teuthology.orchestra.run.smithi179.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:25:05.894 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:25:05.894 INFO:teuthology.task.kernel:Updating grub on smithi179 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:25:05.894 DEBUG:teuthology.orchestra.run.smithi179:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:25:06.518 INFO:teuthology.orchestra.run.smithi179.stderr:Generating grub configuration file ... 2024-03-29T10:25:07.826 INFO:teuthology.orchestra.run.smithi179.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:25:07.868 INFO:teuthology.orchestra.run.smithi179.stderr:done 2024-03-29T10:25:07.871 DEBUG:teuthology.orchestra.run.smithi179:> mktemp 2024-03-29T10:25:07.886 INFO:teuthology.orchestra.run.smithi179.stdout:/tmp/tmp.hYjwqKn8p8 2024-03-29T10:25:07.886 DEBUG:teuthology.orchestra.run.smithi179:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.hYjwqKn8p8 2024-03-29T10:25:07.954 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0666 /tmp/tmp.hYjwqKn8p8 2024-03-29T10:25:08.098 DEBUG:teuthology.orchestra.remote:smithi179:/tmp/tmp.hYjwqKn8p8 is 6KB 2024-03-29T10:25:08.147 DEBUG:teuthology.orchestra.run.smithi179:> rm -fr /tmp/tmp.hYjwqKn8p8 2024-03-29T10:25:08.161 DEBUG:teuthology.orchestra.run.smithi179:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:25:08.226 INFO:teuthology.orchestra.run.smithi179.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:25:08.226 INFO:teuthology.orchestra.run.smithi179.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:25:08.226 INFO:teuthology.orchestra.run.smithi179.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:25:08.227 INFO:teuthology.orchestra.run.smithi179.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:25:08.228 DEBUG:teuthology.orchestra.run.smithi179:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:25:08.356 DEBUG:teuthology.orchestra.run.smithi179:> sudo shutdown -r now 2024-03-29T10:25:38.360 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T10:25:38.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-03-29T10:25:38.361 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:25:56.766 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.179 2024-03-29T10:26:05.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-03-29T10:26:05.769 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:26:08.830 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.179 2024-03-29T10:26:20.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-03-29T10:26:20.832 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:26:35.195 INFO:teuthology.orchestra.run.smithi057.stdout:Extra Packages for Enterprise Linux 162 kB/s | 21 MB 02:10 2024-03-29T10:26:40.262 INFO:teuthology.orchestra.run.smithi057.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-03-29T10:26:42.661 INFO:teuthology.orchestra.run.smithi057.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:26:42.661 INFO:teuthology.orchestra.run.smithi057.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:26:42.764 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:26:42.770 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:26:42.770 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2024-03-29T10:26:42.770 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:26:42.770 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-03-29T10:26:42.770 INFO:teuthology.orchestra.run.smithi057.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:26:42.770 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2024-03-29T10:26:42.770 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:26:42.771 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:26:42.771 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:26:42.771 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:26:42.771 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-03-29T10:26:42.771 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:26:42.771 INFO:teuthology.orchestra.run.smithi057.stdout:Install 4 Packages 2024-03-29T10:26:42.771 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:26:42.771 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 76 M 2024-03-29T10:26:42.772 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 123 M 2024-03-29T10:26:42.772 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-03-29T10:26:42.963 INFO:teuthology.orchestra.run.smithi057.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.0 MB/s | 192 kB 00:00 2024-03-29T10:26:43.464 INFO:teuthology.orchestra.run.smithi057.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 22 MB/s | 15 MB 00:00 2024-03-29T10:26:44.056 INFO:teuthology.orchestra.run.smithi057.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 26 MB/s | 34 MB 00:01 2024-03-29T10:26:44.414 INFO:teuthology.orchestra.run.smithi057.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 19 MB/s | 28 MB 00:01 2024-03-29T10:26:44.417 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:26:44.417 INFO:teuthology.orchestra.run.smithi057.stdout:Total 46 MB/s | 76 MB 00:01 2024-03-29T10:26:44.418 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-03-29T10:26:44.852 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-03-29T10:26:44.852 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-03-29T10:26:45.710 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-03-29T10:26:45.710 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-03-29T10:26:48.221 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-03-29T10:26:48.844 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:26:48.922 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:26:51.798 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:26:51.933 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:26:52.786 DEBUG:teuthology.orchestra.run.smithi179:> true 2024-03-29T10:26:53.286 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-03-29T10:26:53.287 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:26:53.287 DEBUG:teuthology.orchestra.run.smithi179:> uname -r 2024-03-29T10:26:53.302 INFO:teuthology.orchestra.run.smithi179.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:26:53.302 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:26:53.303 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:26:53.303 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:26:54.304 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T10:26:54.304 DEBUG:teuthology.orchestra.run.smithi179:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:26:54.406 INFO:teuthology.orchestra.run.smithi179.stdout:ttyS1 2024-03-29T10:26:54.446 DEBUG:teuthology.parallel:result is None 2024-03-29T10:26:57.019 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:26:57.057 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:27:02.154 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:27:32.912 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:27:32.912 INFO:teuthology.orchestra.run.smithi057.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:27:32.912 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:27:32.956 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:27:33.299 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:27:33.299 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:27:33.299 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:27:33.299 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:27:33.701 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:27:33.701 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:27:33.701 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-03-29T10:27:33.701 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:27:33.702 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:27:33.702 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:27:33.702 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:27:33.702 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:27:33.702 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:27:34.049 DEBUG:teuthology.orchestra.run.smithi057:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:27:34.157 INFO:teuthology.orchestra.run.smithi057.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:27:34.157 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:27:34.158 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:27:34.158 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:27:34.158 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:27:34.158 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T10:27:34.158 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi057.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:27:34.158 DEBUG:teuthology.orchestra.run.smithi057:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:27:34.240 INFO:teuthology.orchestra.run.smithi057.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:27:34.241 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:27:34.241 DEBUG:teuthology.orchestra.run.smithi057:> sudo rpm -qi grub2-tools 2024-03-29T10:27:34.278 INFO:teuthology.orchestra.run.smithi057.stdout:Name : grub2-tools 2024-03-29T10:27:34.278 INFO:teuthology.orchestra.run.smithi057.stdout:Epoch : 1 2024-03-29T10:27:34.278 INFO:teuthology.orchestra.run.smithi057.stdout:Version : 2.06 2024-03-29T10:27:34.279 INFO:teuthology.orchestra.run.smithi057.stdout:Release : 68.el9 2024-03-29T10:27:34.279 INFO:teuthology.orchestra.run.smithi057.stdout:Architecture: x86_64 2024-03-29T10:27:34.279 INFO:teuthology.orchestra.run.smithi057.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:27:34.279 INFO:teuthology.orchestra.run.smithi057.stdout:Group : Unspecified 2024-03-29T10:27:34.279 INFO:teuthology.orchestra.run.smithi057.stdout:Size : 8264254 2024-03-29T10:27:34.279 INFO:teuthology.orchestra.run.smithi057.stdout:License : GPLv3+ 2024-03-29T10:27:34.279 INFO:teuthology.orchestra.run.smithi057.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:27:34.279 INFO:teuthology.orchestra.run.smithi057.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:27:34.279 INFO:teuthology.orchestra.run.smithi057.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:27:34.279 INFO:teuthology.orchestra.run.smithi057.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:27:34.279 INFO:teuthology.orchestra.run.smithi057.stdout:Packager : builder@centos.org 2024-03-29T10:27:34.279 INFO:teuthology.orchestra.run.smithi057.stdout:Vendor : CentOS 2024-03-29T10:27:34.279 INFO:teuthology.orchestra.run.smithi057.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:27:34.279 INFO:teuthology.orchestra.run.smithi057.stdout:Summary : Support tools for GRUB. 2024-03-29T10:27:34.279 INFO:teuthology.orchestra.run.smithi057.stdout:Description : 2024-03-29T10:27:34.280 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:27:34.280 INFO:teuthology.orchestra.run.smithi057.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:27:34.280 INFO:teuthology.orchestra.run.smithi057.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:27:34.280 INFO:teuthology.orchestra.run.smithi057.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:27:34.280 INFO:teuthology.orchestra.run.smithi057.stdout:hardware devices. 2024-03-29T10:27:34.280 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:27:34.280 INFO:teuthology.orchestra.run.smithi057.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:27:34.281 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:27:34.281 INFO:teuthology.task.kernel:Updating grub on smithi057 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:27:34.281 DEBUG:teuthology.orchestra.run.smithi057:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:27:34.883 INFO:teuthology.orchestra.run.smithi057.stderr:Generating grub configuration file ... 2024-03-29T10:27:36.350 INFO:teuthology.orchestra.run.smithi057.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:27:36.383 INFO:teuthology.orchestra.run.smithi057.stderr:done 2024-03-29T10:27:36.385 DEBUG:teuthology.orchestra.run.smithi057:> mktemp 2024-03-29T10:27:36.401 INFO:teuthology.orchestra.run.smithi057.stdout:/tmp/tmp.Y3d97QkrMZ 2024-03-29T10:27:36.401 DEBUG:teuthology.orchestra.run.smithi057:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Y3d97QkrMZ 2024-03-29T10:27:36.473 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0666 /tmp/tmp.Y3d97QkrMZ 2024-03-29T10:27:36.615 DEBUG:teuthology.orchestra.remote:smithi057:/tmp/tmp.Y3d97QkrMZ is 6KB 2024-03-29T10:27:36.667 DEBUG:teuthology.orchestra.run.smithi057:> rm -fr /tmp/tmp.Y3d97QkrMZ 2024-03-29T10:27:36.682 DEBUG:teuthology.orchestra.run.smithi057:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:27:36.752 INFO:teuthology.orchestra.run.smithi057.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:27:36.753 INFO:teuthology.orchestra.run.smithi057.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:27:36.753 INFO:teuthology.orchestra.run.smithi057.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:27:36.753 INFO:teuthology.orchestra.run.smithi057.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:27:36.754 DEBUG:teuthology.orchestra.run.smithi057:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:27:36.921 DEBUG:teuthology.orchestra.run.smithi057:> sudo shutdown -r now 2024-03-29T10:28:06.924 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T10:28:06.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-03-29T10:28:06.926 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:29:06.930 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T10:29:15.931 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-03-29T10:29:15.932 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:29:23.361 DEBUG:teuthology.orchestra.run.smithi057:> true 2024-03-29T10:29:23.828 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-03-29T10:29:23.828 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:29:23.828 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2024-03-29T10:29:23.844 INFO:teuthology.orchestra.run.smithi057.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:29:23.845 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:29:23.845 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:29:23.845 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:29:24.845 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T10:29:24.846 DEBUG:teuthology.orchestra.run.smithi057:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:29:24.911 INFO:teuthology.orchestra.run.smithi057.stdout:ttyS1 2024-03-29T10:29:24.960 DEBUG:teuthology.parallel:result is None 2024-03-29T10:29:24.961 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T10:29:24.973 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T10:29:24.973 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:29:24.977 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:29:24.999 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T10:29:25.008 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T10:29:25.042 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T10:29:25.042 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:29:25.045 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:29:25.130 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T10:29:25.140 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T10:29:25.140 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:29:25.146 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:29:25.181 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:29:25.195 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:29:25.196 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:29:25.209 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:29:25.211 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T10:29:25.218 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T10:29:25.218 DEBUG:teuthology.orchestra.run.smithi057:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:29:25.241 DEBUG:teuthology.orchestra.run.smithi179:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:29:25.271 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T10:29:25.280 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T10:29:25.281 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:29:25.309 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:29:25.327 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:29:25.402 DEBUG:teuthology.orchestra.run.smithi057:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:29:25.447 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:29:25.527 DEBUG:teuthology.orchestra.run.smithi057:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:29:25.570 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-29T10:29:25.570 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:29:25.645 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:29:25.687 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:29:25.741 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:29:25.826 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:29:25.871 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T10:29:25.871 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:29:25.941 DEBUG:teuthology.orchestra.run.smithi057:> sudo service rsyslog restart 2024-03-29T10:29:25.944 DEBUG:teuthology.orchestra.run.smithi179:> sudo service rsyslog restart 2024-03-29T10:29:25.982 INFO:teuthology.orchestra.run.smithi057.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:29:26.037 INFO:teuthology.orchestra.run.smithi179.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:29:26.543 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T10:29:26.550 INFO:teuthology.task.internal:Starting timer... 2024-03-29T10:29:26.551 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T10:29:26.586 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T10:29:26.618 DEBUG:teuthology.orchestra.run.smithi057:> sudo service auditd rotate 2024-03-29T10:29:26.696 INFO:teuthology.orchestra.run.smithi057.stdout:Rotating logs: 2024-03-29T10:29:26.698 DEBUG:teuthology.orchestra.run.smithi179:> sudo service auditd rotate 2024-03-29T10:29:26.771 INFO:teuthology.orchestra.run.smithi179.stdout:Rotating logs: 2024-03-29T10:29:26.774 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T10:29:26.774 DEBUG:teuthology.orchestra.run.smithi057:> /usr/sbin/getenforce 2024-03-29T10:29:26.803 INFO:teuthology.orchestra.run.smithi057.stdout:Permissive 2024-03-29T10:29:26.803 DEBUG:teuthology.orchestra.run.smithi179:> /usr/sbin/getenforce 2024-03-29T10:29:26.841 INFO:teuthology.orchestra.run.smithi179.stdout:Permissive 2024-03-29T10:29:26.842 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi057.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi179.front.sepia.ceph.com': 'permissive'} 2024-03-29T10:29:26.842 DEBUG:teuthology.orchestra.run.smithi057:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:29:26.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:29:26.871 DEBUG:teuthology.orchestra.run.smithi179:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:29:26.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:29:26.911 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T10:29:26.912 DEBUG:teuthology.orchestra.run.smithi057:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:29:26.955 DEBUG:teuthology.orchestra.run.smithi179:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:29:26.994 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T10:29:27.004 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T10:29:27.022 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T10:29:27.023 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi057.front.sepia.ceph.com,smithi179.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T10:33:48.667 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi057.front.sepia.ceph.com'), Remote(name='ubuntu@smithi179.front.sepia.ceph.com')] 2024-03-29T10:33:48.668 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-03-29T10:33:48.669 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:33:48.761 DEBUG:teuthology.orchestra.run.smithi057:> true 2024-03-29T10:33:48.798 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-03-29T10:33:48.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-03-29T10:33:48.799 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:33:48.891 DEBUG:teuthology.orchestra.run.smithi179:> true 2024-03-29T10:33:48.961 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-03-29T10:33:48.961 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T10:33:48.975 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T10:33:48.976 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:33:48.976 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:33:48.978 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:33:48.979 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:33:49.008 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:33:49.025 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:33:49.033 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:33:49.050 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:33:49.055 INFO:teuthology.orchestra.run.smithi057.stderr:sudo: ntpd: command not found 2024-03-29T10:33:49.068 INFO:teuthology.orchestra.run.smithi057.stdout:506 Cannot talk to daemon 2024-03-29T10:33:49.079 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: ntpd: command not found 2024-03-29T10:33:49.085 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:33:49.093 INFO:teuthology.orchestra.run.smithi179.stdout:506 Cannot talk to daemon 2024-03-29T10:33:49.101 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:33:49.109 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:33:49.126 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:33:49.150 INFO:teuthology.orchestra.run.smithi057.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:33:49.153 INFO:teuthology.orchestra.run.smithi057.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:33:49.153 INFO:teuthology.orchestra.run.smithi057.stdout:=============================================================================== 2024-03-29T10:33:49.153 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:33:49.153 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:33:49.153 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:33:49.153 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:33:49.181 INFO:teuthology.orchestra.run.smithi179.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:33:49.184 INFO:teuthology.orchestra.run.smithi179.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:33:49.184 INFO:teuthology.orchestra.run.smithi179.stdout:=============================================================================== 2024-03-29T10:33:49.184 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:33:49.184 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:33:49.185 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:33:49.185 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:33:49.186 INFO:teuthology.run_tasks:Running task install... 2024-03-29T10:33:49.194 DEBUG:teuthology.task.install:project ceph 2024-03-29T10:33:49.194 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T10:33:49.194 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T10:33:49.194 INFO:teuthology.task.install:Using flavor: default 2024-03-29T10:33:49.200 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T10:33:49.200 INFO:teuthology.task.install:extra packages: [] 2024-03-29T10:33:49.201 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:33:49.201 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:33:49.203 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:33:49.204 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:33:49.352 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:33:49.352 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:33:49.356 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:33:49.356 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:33:49.469 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:33:49.470 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T10:33:49.470 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:33:49.498 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:33:49.498 DEBUG:teuthology.orchestra.run.smithi179:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:33:49.566 DEBUG:teuthology.orchestra.run.smithi179:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:33:49.605 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:33:49.606 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-29T10:33:49.606 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:33:49.634 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:33:49.635 DEBUG:teuthology.orchestra.run.smithi057:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:33:49.645 DEBUG:teuthology.orchestra.run.smithi179:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:33:49.704 DEBUG:teuthology.orchestra.run.smithi057:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:33:49.711 INFO:teuthology.orchestra.run.smithi179.stdout:check_obsoletes = 1 2024-03-29T10:33:49.712 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum clean all 2024-03-29T10:33:49.784 DEBUG:teuthology.orchestra.run.smithi057:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:33:49.851 INFO:teuthology.orchestra.run.smithi057.stdout:check_obsoletes = 1 2024-03-29T10:33:49.853 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum clean all 2024-03-29T10:33:49.949 INFO:teuthology.orchestra.run.smithi179.stdout:74 files removed 2024-03-29T10:33:49.977 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-radosgw 2024-03-29T10:33:50.102 INFO:teuthology.orchestra.run.smithi057.stdout:74 files removed 2024-03-29T10:33:50.129 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-radosgw 2024-03-29T10:33:50.586 INFO:teuthology.orchestra.run.smithi179.stdout:ceph packages for x86_64 265 kB/s | 82 kB 00:00 2024-03-29T10:33:51.389 INFO:teuthology.orchestra.run.smithi057.stdout:ceph packages for x86_64 86 kB/s | 82 kB 00:00 2024-03-29T10:33:51.478 INFO:teuthology.orchestra.run.smithi179.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-29T10:33:51.765 INFO:teuthology.orchestra.run.smithi057.stdout:ceph noarch packages 31 kB/s | 10 kB 00:00 2024-03-29T10:33:51.937 INFO:teuthology.orchestra.run.smithi179.stdout:ceph source packages 4.5 kB/s | 1.9 kB 00:00 2024-03-29T10:33:52.166 INFO:teuthology.orchestra.run.smithi057.stdout:ceph source packages 5.3 kB/s | 1.9 kB 00:00 2024-03-29T10:33:52.432 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-29T10:33:52.801 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-29T10:33:53.923 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-03-29T10:33:54.277 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-29T10:33:55.008 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-29T10:33:55.159 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-29T10:33:56.732 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - AppStream 30 MB/s | 19 MB 00:00 2024-03-29T10:34:00.666 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-29T10:34:02.700 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 16 kB 00:00 2024-03-29T10:34:03.769 INFO:teuthology.orchestra.run.smithi179.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-29T10:34:08.744 INFO:teuthology.orchestra.run.smithi179.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-29T10:34:11.336 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:34:11.353 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:34:11.353 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-29T10:34:11.353 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:34:11.353 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T10:34:11.353 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:34:11.353 INFO:teuthology.orchestra.run.smithi179.stdout:Upgrading: 2024-03-29T10:34:11.354 INFO:teuthology.orchestra.run.smithi179.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:34:11.354 INFO:teuthology.orchestra.run.smithi179.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:34:11.354 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-29T10:34:11.354 INFO:teuthology.orchestra.run.smithi179.stdout: boost-program-options 2024-03-29T10:34:11.354 INFO:teuthology.orchestra.run.smithi179.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:34:11.354 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:34:11.354 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:34:11.354 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:34:11.354 INFO:teuthology.orchestra.run.smithi179.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:34:11.354 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:34:11.354 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:34:11.354 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:34:11.354 INFO:teuthology.orchestra.run.smithi179.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:34:11.355 INFO:teuthology.orchestra.run.smithi179.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:34:11.355 INFO:teuthology.orchestra.run.smithi179.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:34:11.355 INFO:teuthology.orchestra.run.smithi179.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:34:11.355 INFO:teuthology.orchestra.run.smithi179.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:34:11.355 INFO:teuthology.orchestra.run.smithi179.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:34:11.355 INFO:teuthology.orchestra.run.smithi179.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:34:11.355 INFO:teuthology.orchestra.run.smithi179.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:34:11.355 INFO:teuthology.orchestra.run.smithi179.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:34:11.356 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-argparse 2024-03-29T10:34:11.356 INFO:teuthology.orchestra.run.smithi179.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:34:11.356 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:34:11.356 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:34:11.356 INFO:teuthology.orchestra.run.smithi179.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:34:11.356 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:34:11.356 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:34:11.356 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:34:11.356 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:34:11.356 INFO:teuthology.orchestra.run.smithi179.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:34:11.356 INFO:teuthology.orchestra.run.smithi179.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:34:11.356 INFO:teuthology.orchestra.run.smithi179.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:34:11.356 INFO:teuthology.orchestra.run.smithi179.stdout:Installing weak dependencies: 2024-03-29T10:34:11.356 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:34:11.357 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:34:11.357 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T10:34:11.357 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:34:11.357 INFO:teuthology.orchestra.run.smithi179.stdout:Install 30 Packages 2024-03-29T10:34:11.357 INFO:teuthology.orchestra.run.smithi179.stdout:Upgrade 2 Packages 2024-03-29T10:34:11.357 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:34:11.357 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 58 M 2024-03-29T10:34:11.357 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T10:34:12.853 INFO:teuthology.orchestra.run.smithi179.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 8.5 MB/s | 9.7 MB 00:01 2024-03-29T10:34:12.936 INFO:teuthology.orchestra.run.smithi179.stdout:(2/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 4.4 MB/s | 5.3 MB 00:01 2024-03-29T10:34:12.961 INFO:teuthology.orchestra.run.smithi179.stdout:(3/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 232 kB/s | 25 kB 00:00 2024-03-29T10:34:13.070 INFO:teuthology.orchestra.run.smithi179.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 4.5 MB/s | 498 kB 00:00 2024-03-29T10:34:13.096 INFO:teuthology.orchestra.run.smithi179.stdout:(5/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 4.6 MB/s | 742 kB 00:00 2024-03-29T10:34:13.162 INFO:teuthology.orchestra.run.smithi179.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 678 kB/s | 45 kB 00:00 2024-03-29T10:34:13.254 INFO:teuthology.orchestra.run.smithi179.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.4 MB/s | 131 kB 00:00 2024-03-29T10:34:13.347 INFO:teuthology.orchestra.run.smithi179.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.7 MB/s | 162 kB 00:00 2024-03-29T10:34:13.455 INFO:teuthology.orchestra.run.smithi179.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 2.9 MB/s | 320 kB 00:00 2024-03-29T10:34:13.539 INFO:teuthology.orchestra.run.smithi179.stdout:(10/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 11 MB/s | 5.1 MB 00:00 2024-03-29T10:34:13.564 INFO:teuthology.orchestra.run.smithi179.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 2.7 MB/s | 298 kB 00:00 2024-03-29T10:34:13.598 INFO:teuthology.orchestra.run.smithi179.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 99 kB 00:00 2024-03-29T10:34:13.714 INFO:teuthology.orchestra.run.smithi179.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 704 kB/s | 106 kB 00:00 2024-03-29T10:34:13.740 INFO:teuthology.orchestra.run.smithi179.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-03-29T10:34:13.765 INFO:teuthology.orchestra.run.smithi179.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-29T10:34:13.999 INFO:teuthology.orchestra.run.smithi179.stdout:(16/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 8.7 MB/s | 20 MB 00:02 2024-03-29T10:34:14.040 INFO:teuthology.orchestra.run.smithi179.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.1 MB/s | 661 kB 00:00 2024-03-29T10:34:14.066 INFO:teuthology.orchestra.run.smithi179.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 941 kB/s | 282 kB 00:00 2024-03-29T10:34:14.091 INFO:teuthology.orchestra.run.smithi179.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 452 kB/s | 41 kB 00:00 2024-03-29T10:34:14.116 INFO:teuthology.orchestra.run.smithi179.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 220 kB 00:00 2024-03-29T10:34:14.142 INFO:teuthology.orchestra.run.smithi179.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 696 kB/s | 35 kB 00:00 2024-03-29T10:34:14.167 INFO:teuthology.orchestra.run.smithi179.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 617 kB/s | 46 kB 00:00 2024-03-29T10:34:14.475 INFO:teuthology.orchestra.run.smithi179.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 84 kB/s | 26 kB 00:00 2024-03-29T10:34:14.600 INFO:teuthology.orchestra.run.smithi179.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 638 kB/s | 309 kB 00:00 2024-03-29T10:34:14.626 INFO:teuthology.orchestra.run.smithi179.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 322 kB/s | 49 kB 00:00 2024-03-29T10:34:14.684 INFO:teuthology.orchestra.run.smithi179.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 805 kB/s | 67 kB 00:00 2024-03-29T10:34:14.785 INFO:teuthology.orchestra.run.smithi179.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-29T10:34:14.902 INFO:teuthology.orchestra.run.smithi179.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 6.0 MB/s | 4.4 MB 00:00 2024-03-29T10:34:14.935 INFO:teuthology.orchestra.run.smithi179.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.6 MB/s | 838 kB 00:00 2024-03-29T10:34:15.027 INFO:teuthology.orchestra.run.smithi179.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 6.5 MB/s | 1.6 MB 00:00 2024-03-29T10:34:15.319 INFO:teuthology.orchestra.run.smithi179.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 8.2 MB/s | 3.1 MB 00:00 2024-03-29T10:34:15.419 INFO:teuthology.orchestra.run.smithi179.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 6.5 MB/s | 3.4 MB 00:00 2024-03-29T10:34:15.426 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:34:15.426 INFO:teuthology.orchestra.run.smithi179.stdout:Total 14 MB/s | 58 MB 00:04 2024-03-29T10:34:15.427 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T10:34:15.462 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T10:34:15.462 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T10:34:15.944 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T10:34:15.944 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T10:34:17.257 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - AppStream 923 kB/s | 19 MB 00:20 2024-03-29T10:34:17.402 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T10:34:17.471 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:34:17.507 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:34:17.533 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:34:17.555 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:34:17.572 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:34:17.757 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:34:17.812 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:34:18.158 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:34:18.226 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:34:18.310 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:34:18.400 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:34:18.445 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:34:18.507 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:34:18.535 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:34:18.579 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:34:18.648 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:34:18.671 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:34:18.735 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:34:18.770 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:34:18.879 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:34:18.912 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:34:18.953 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:34:18.985 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:34:19.014 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:34:19.047 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:34:19.378 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:34:19.411 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:34:19.500 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:34:19.532 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:34:19.571 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:34:20.028 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:34:20.054 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:34:20.126 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:34:20.193 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:34:20.223 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:34:22.336 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:34:22.378 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:34:22.936 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:34:22.990 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:34:23.060 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:34:23.060 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:34:23.060 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:34:23.060 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:34:23.099 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:34:28.916 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - CRB 806 kB/s | 6.1 MB 00:07 2024-03-29T10:34:31.300 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-29T10:34:32.634 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:34:32.634 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /sys 2024-03-29T10:34:32.634 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /proc 2024-03-29T10:34:32.634 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /mnt 2024-03-29T10:34:32.634 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /var/tmp 2024-03-29T10:34:32.634 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /home 2024-03-29T10:34:32.634 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /root 2024-03-29T10:34:32.634 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /tmp 2024-03-29T10:34:32.635 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:34:33.554 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:34:33.579 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:34:33.579 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:34:33.579 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:34:33.579 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:34:33.579 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:34:33.579 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:34:33.581 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:34:33.665 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:34:33.665 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:34:34.847 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:34:34.847 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:34:34.847 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:34:34.848 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:34:34.848 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:34:34.848 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:34:34.848 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:34:34.848 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:34:34.848 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:34:34.848 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:34:34.849 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:34:34.849 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:34:34.849 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:34:34.849 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:34:34.849 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:34:34.849 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:34:34.849 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:34:34.849 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:34:34.849 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:34:34.849 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:34:34.849 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:34:34.849 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:34:34.849 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:34:34.849 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:34:34.850 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:34:34.850 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:34:34.850 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:34:34.850 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:34:34.850 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:34:34.850 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:34:34.850 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:34:34.850 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:34:34.850 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:34:34.850 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:34:35.738 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:34:35.738 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:34:35.738 INFO:teuthology.orchestra.run.smithi179.stdout:Upgraded: 2024-03-29T10:34:35.738 INFO:teuthology.orchestra.run.smithi179.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:35.738 INFO:teuthology.orchestra.run.smithi179.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:35.738 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T10:34:35.738 INFO:teuthology.orchestra.run.smithi179.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:34:35.739 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:35.739 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:35.739 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:35.739 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:35.739 INFO:teuthology.orchestra.run.smithi179.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:34:35.739 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:34:35.739 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:34:35.739 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:35.739 INFO:teuthology.orchestra.run.smithi179.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:34:35.739 INFO:teuthology.orchestra.run.smithi179.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:34:35.739 INFO:teuthology.orchestra.run.smithi179.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:34:35.739 INFO:teuthology.orchestra.run.smithi179.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:35.739 INFO:teuthology.orchestra.run.smithi179.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:34:35.739 INFO:teuthology.orchestra.run.smithi179.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:35.739 INFO:teuthology.orchestra.run.smithi179.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:34:35.740 INFO:teuthology.orchestra.run.smithi179.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:34:35.740 INFO:teuthology.orchestra.run.smithi179.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:34:35.740 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:35.740 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:35.740 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:35.740 INFO:teuthology.orchestra.run.smithi179.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:34:35.740 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:35.740 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:35.740 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:35.740 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:34:35.740 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:34:35.740 INFO:teuthology.orchestra.run.smithi179.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:34:35.740 INFO:teuthology.orchestra.run.smithi179.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:34:35.740 INFO:teuthology.orchestra.run.smithi179.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:34:35.741 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:34:35.741 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:34:36.002 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-test 2024-03-29T10:34:36.553 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:00:28 ago on Fri 29 Mar 2024 10:34:08 AM UTC. 2024-03-29T10:34:36.702 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:34:36.702 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:34:36.702 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-29T10:34:36.702 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:34:36.703 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T10:34:36.703 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:34:36.703 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-29T10:34:36.703 INFO:teuthology.orchestra.run.smithi179.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:34:36.703 INFO:teuthology.orchestra.run.smithi179.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:34:36.703 INFO:teuthology.orchestra.run.smithi179.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:34:36.703 INFO:teuthology.orchestra.run.smithi179.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:34:36.703 INFO:teuthology.orchestra.run.smithi179.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:34:36.703 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:34:36.703 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T10:34:36.703 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:34:36.703 INFO:teuthology.orchestra.run.smithi179.stdout:Install 6 Packages 2024-03-29T10:34:36.703 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:34:36.704 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 48 M 2024-03-29T10:34:36.704 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 204 M 2024-03-29T10:34:36.704 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T10:34:36.913 INFO:teuthology.orchestra.run.smithi179.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T10:34:36.963 INFO:teuthology.orchestra.run.smithi179.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T10:34:36.989 INFO:teuthology.orchestra.run.smithi179.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-03-29T10:34:37.123 INFO:teuthology.orchestra.run.smithi179.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 440 kB/s | 163 kB 00:00 2024-03-29T10:34:37.489 INFO:teuthology.orchestra.run.smithi179.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 374 kB/s | 187 kB 00:00 2024-03-29T10:34:39.256 INFO:teuthology.orchestra.run.smithi179.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 19 MB/s | 47 MB 00:02 2024-03-29T10:34:39.259 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:34:39.260 INFO:teuthology.orchestra.run.smithi179.stdout:Total 19 MB/s | 48 MB 00:02 2024-03-29T10:34:39.274 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T10:34:39.294 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T10:34:39.294 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T10:34:39.677 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T10:34:39.677 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T10:34:40.296 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T10:34:40.373 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:34:40.411 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:34:40.476 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:34:40.515 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:34:40.542 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:34:45.027 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:34:45.100 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:34:47.568 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:34:47.568 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:34:47.568 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:34:47.568 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:34:47.568 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:34:47.568 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:34:47.942 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:34:47.942 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:34:47.942 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T10:34:47.942 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:47.942 INFO:teuthology.orchestra.run.smithi179.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:34:47.943 INFO:teuthology.orchestra.run.smithi179.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:47.943 INFO:teuthology.orchestra.run.smithi179.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:34:47.943 INFO:teuthology.orchestra.run.smithi179.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:34:47.943 INFO:teuthology.orchestra.run.smithi179.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:34:47.943 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:34:47.943 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:34:48.297 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph 2024-03-29T10:34:48.875 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:00:40 ago on Fri 29 Mar 2024 10:34:08 AM UTC. 2024-03-29T10:34:49.037 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:34:49.039 INFO:teuthology.orchestra.run.smithi179.stdout:========================================================================================= 2024-03-29T10:34:49.039 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-29T10:34:49.039 INFO:teuthology.orchestra.run.smithi179.stdout:========================================================================================= 2024-03-29T10:34:49.039 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T10:34:49.040 INFO:teuthology.orchestra.run.smithi179.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:34:49.040 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-29T10:34:49.040 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:34:49.040 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:34:49.040 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:34:49.040 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:34:49.040 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:34:49.040 INFO:teuthology.orchestra.run.smithi179.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:34:49.040 INFO:teuthology.orchestra.run.smithi179.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:34:49.040 INFO:teuthology.orchestra.run.smithi179.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:34:49.040 INFO:teuthology.orchestra.run.smithi179.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:34:49.041 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:34:49.041 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:34:49.041 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:34:49.041 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:34:49.041 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:34:49.041 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:34:49.041 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:34:49.041 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:34:49.041 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:34:49.041 INFO:teuthology.orchestra.run.smithi179.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:34:49.041 INFO:teuthology.orchestra.run.smithi179.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:34:49.041 INFO:teuthology.orchestra.run.smithi179.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:34:49.041 INFO:teuthology.orchestra.run.smithi179.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:34:49.042 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:34:49.042 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:34:49.042 INFO:teuthology.orchestra.run.smithi179.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:34:49.042 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:34:49.042 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:34:49.042 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:34:49.042 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:34:49.042 INFO:teuthology.orchestra.run.smithi179.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:34:49.042 INFO:teuthology.orchestra.run.smithi179.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:34:49.042 INFO:teuthology.orchestra.run.smithi179.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:34:49.042 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:34:49.042 INFO:teuthology.orchestra.run.smithi179.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:34:49.043 INFO:teuthology.orchestra.run.smithi179.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:34:49.043 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:34:49.043 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T10:34:49.043 INFO:teuthology.orchestra.run.smithi179.stdout:========================================================================================= 2024-03-29T10:34:49.043 INFO:teuthology.orchestra.run.smithi179.stdout:Install 36 Packages 2024-03-29T10:34:49.043 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:34:49.043 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 30 M 2024-03-29T10:34:49.044 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 107 M 2024-03-29T10:34:49.044 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T10:34:49.775 INFO:teuthology.orchestra.run.smithi179.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 38 kB/s | 6.6 kB 00:00 2024-03-29T10:34:49.984 INFO:teuthology.orchestra.run.smithi179.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 3.9 MB/s | 1.5 MB 00:00 2024-03-29T10:34:50.209 INFO:teuthology.orchestra.run.smithi179.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 11 MB/s | 4.7 MB 00:00 2024-03-29T10:34:50.268 INFO:teuthology.orchestra.run.smithi179.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 4.2 MB/s | 247 kB 00:00 2024-03-29T10:34:50.385 INFO:teuthology.orchestra.run.smithi179.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-29T10:34:50.427 INFO:teuthology.orchestra.run.smithi179.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T10:34:50.460 INFO:teuthology.orchestra.run.smithi179.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 946 kB/s | 32 kB 00:00 2024-03-29T10:34:50.502 INFO:teuthology.orchestra.run.smithi179.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-29T10:34:50.536 INFO:teuthology.orchestra.run.smithi179.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T10:34:50.578 INFO:teuthology.orchestra.run.smithi179.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-29T10:34:50.653 INFO:teuthology.orchestra.run.smithi179.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T10:34:50.687 INFO:teuthology.orchestra.run.smithi179.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T10:34:50.737 INFO:teuthology.orchestra.run.smithi179.stdout:(13/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86 1.9 MB/s | 2.1 MB 00:01 2024-03-29T10:34:50.762 INFO:teuthology.orchestra.run.smithi179.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 1.6 MB/s | 124 kB 00:00 2024-03-29T10:34:50.788 INFO:teuthology.orchestra.run.smithi179.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T10:34:50.888 INFO:teuthology.orchestra.run.smithi179.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-03-29T10:34:51.055 INFO:teuthology.orchestra.run.smithi179.stdout:(17/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 16 MB/s | 17 MB 00:01 2024-03-29T10:34:51.121 INFO:teuthology.orchestra.run.smithi179.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 348 kB/s | 125 kB 00:00 2024-03-29T10:34:51.206 INFO:teuthology.orchestra.run.smithi179.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 289 kB/s | 43 kB 00:00 2024-03-29T10:34:51.264 INFO:teuthology.orchestra.run.smithi179.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 402 kB/s | 151 kB 00:00 2024-03-29T10:34:51.323 INFO:teuthology.orchestra.run.smithi179.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-03-29T10:34:51.356 INFO:teuthology.orchestra.run.smithi179.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 733 kB/s | 172 kB 00:00 2024-03-29T10:34:51.381 INFO:teuthology.orchestra.run.smithi179.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 303 kB/s | 18 kB 00:00 2024-03-29T10:34:51.415 INFO:teuthology.orchestra.run.smithi179.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-03-29T10:34:51.440 INFO:teuthology.orchestra.run.smithi179.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 276 kB/s | 23 kB 00:00 2024-03-29T10:34:51.466 INFO:teuthology.orchestra.run.smithi179.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 232 kB/s | 19 kB 00:00 2024-03-29T10:34:51.491 INFO:teuthology.orchestra.run.smithi179.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-03-29T10:34:51.516 INFO:teuthology.orchestra.run.smithi179.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-03-29T10:34:51.542 INFO:teuthology.orchestra.run.smithi179.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-29T10:34:51.575 INFO:teuthology.orchestra.run.smithi179.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-29T10:34:51.602 INFO:teuthology.orchestra.run.smithi179.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 196 kB/s | 16 kB 00:00 2024-03-29T10:34:51.626 INFO:teuthology.orchestra.run.smithi179.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2024-03-29T10:34:51.651 INFO:teuthology.orchestra.run.smithi179.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 473 kB/s | 36 kB 00:00 2024-03-29T10:34:51.693 INFO:teuthology.orchestra.run.smithi179.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-29T10:34:51.718 INFO:teuthology.orchestra.run.smithi179.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 299 kB/s | 20 kB 00:00 2024-03-29T10:34:51.760 INFO:teuthology.orchestra.run.smithi179.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-03-29T10:34:51.762 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:34:51.762 INFO:teuthology.orchestra.run.smithi179.stdout:Total 11 MB/s | 30 MB 00:02 2024-03-29T10:34:51.788 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T10:34:51.830 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T10:34:51.830 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T10:34:52.239 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T10:34:52.240 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T10:34:53.569 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T10:34:53.646 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:34:53.688 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:34:53.715 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:34:53.777 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:34:53.822 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:34:53.855 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:34:53.946 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:34:54.016 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:34:54.058 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:34:54.129 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:34:54.184 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:34:54.221 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:34:54.270 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:34:54.326 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:34:54.511 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:34:54.559 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:34:54.597 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:34:54.632 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:34:54.662 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:34:54.738 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:34:54.772 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:34:54.805 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:34:54.846 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:34:54.881 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:34:54.916 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:34:55.006 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:34:55.046 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:34:55.119 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:34:55.255 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:34:55.382 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:34:55.538 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:34:55.568 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:34:55.593 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:34:55.594 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:34:55.594 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:34:55.594 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:34:55.594 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:34:55.594 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:34:57.027 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:34:57.053 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:34:57.053 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:34:57.053 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:34:57.053 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:34:57.053 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:34:57.054 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:34:57.495 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:34:57.517 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:34:57.517 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:34:57.517 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:34:57.518 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:34:57.518 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:34:57.518 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:34:57.729 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:34:57.753 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:34:57.754 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:34:57.754 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:34:57.754 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:34:57.754 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:34:57.754 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:34:57.818 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:34:59.680 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:34:59.681 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:34:59.681 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:34:59.681 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:34:59.681 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:34:59.681 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:34:59.681 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:34:59.681 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:34:59.681 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:34:59.681 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:34:59.681 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:34:59.681 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:34:59.682 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:34:59.682 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:34:59.682 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:34:59.682 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:34:59.682 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:34:59.682 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:34:59.682 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:34:59.682 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:34:59.682 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:34:59.682 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:34:59.682 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:34:59.682 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:34:59.682 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:34:59.682 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:34:59.684 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:34:59.684 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:34:59.684 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:34:59.684 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:34:59.684 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:34:59.684 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:34:59.684 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:34:59.684 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:34:59.684 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:34:59.684 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:35:00.644 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:35:00.644 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:00.644 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T10:35:00.644 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:35:00.644 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:35:00.644 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:35:00.644 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:35:00.644 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:35:00.644 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:35:00.645 INFO:teuthology.orchestra.run.smithi179.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:35:00.645 INFO:teuthology.orchestra.run.smithi179.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:35:00.645 INFO:teuthology.orchestra.run.smithi179.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:35:00.645 INFO:teuthology.orchestra.run.smithi179.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:35:00.645 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:35:00.645 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:35:00.645 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:35:00.645 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:35:00.645 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:35:00.645 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:35:00.645 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:35:00.645 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:35:00.645 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:35:00.646 INFO:teuthology.orchestra.run.smithi179.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:35:00.646 INFO:teuthology.orchestra.run.smithi179.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:35:00.646 INFO:teuthology.orchestra.run.smithi179.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:35:00.646 INFO:teuthology.orchestra.run.smithi179.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:35:00.646 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:35:00.646 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:35:00.646 INFO:teuthology.orchestra.run.smithi179.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:35:00.646 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:35:00.646 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:35:00.646 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:35:00.646 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:35:00.646 INFO:teuthology.orchestra.run.smithi179.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:35:00.646 INFO:teuthology.orchestra.run.smithi179.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:35:00.646 INFO:teuthology.orchestra.run.smithi179.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:35:00.647 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:35:00.647 INFO:teuthology.orchestra.run.smithi179.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:35:00.647 INFO:teuthology.orchestra.run.smithi179.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:35:00.647 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:00.647 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:35:00.944 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-base 2024-03-29T10:35:01.511 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:00:53 ago on Fri 29 Mar 2024 10:34:08 AM UTC. 2024-03-29T10:35:01.610 INFO:teuthology.orchestra.run.smithi179.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:35:01.663 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:35:01.664 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-29T10:35:01.664 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:35:01.740 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install cephadm 2024-03-29T10:35:02.295 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:00:54 ago on Fri 29 Mar 2024 10:34:08 AM UTC. 2024-03-29T10:35:02.446 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:35:02.446 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:35:02.446 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-29T10:35:02.447 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:35:02.447 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T10:35:02.447 INFO:teuthology.orchestra.run.smithi179.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:35:02.447 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:02.447 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T10:35:02.447 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:35:02.447 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-29T10:35:02.447 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:02.447 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 747 k 2024-03-29T10:35:02.447 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 754 k 2024-03-29T10:35:02.447 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T10:35:02.737 INFO:teuthology.orchestra.run.smithi179.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.5 MB/s | 747 kB 00:00 2024-03-29T10:35:02.738 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:02.738 INFO:teuthology.orchestra.run.smithi179.stdout:Total 2.5 MB/s | 747 kB 00:00 2024-03-29T10:35:02.738 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T10:35:02.742 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T10:35:02.742 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T10:35:02.748 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T10:35:02.748 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T10:35:02.820 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T10:35:03.327 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:35:03.408 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:35:03.763 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:35:04.039 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:35:04.039 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:04.039 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T10:35:04.039 INFO:teuthology.orchestra.run.smithi179.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:35:04.039 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:04.039 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:35:04.290 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:35:04.864 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Mar 2024 10:34:08 AM UTC. 2024-03-29T10:35:05.017 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:35:05.017 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:35:05.017 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-29T10:35:05.018 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:35:05.018 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T10:35:05.018 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:35:05.018 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:05.018 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T10:35:05.018 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:35:05.018 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-29T10:35:05.018 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:05.018 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 145 k 2024-03-29T10:35:05.018 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 436 k 2024-03-29T10:35:05.018 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T10:35:05.584 INFO:teuthology.orchestra.run.smithi179.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 257 kB/s | 145 kB 00:00 2024-03-29T10:35:05.585 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:05.585 INFO:teuthology.orchestra.run.smithi179.stdout:Total 256 kB/s | 145 kB 00:00 2024-03-29T10:35:05.585 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T10:35:05.592 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T10:35:05.593 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T10:35:05.646 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T10:35:05.646 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T10:35:05.784 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T10:35:05.851 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:35:05.877 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:35:05.877 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:35:05.877 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:35:05.877 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:06.890 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:35:06.890 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:06.891 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T10:35:06.891 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:35:06.891 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:06.891 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:35:07.141 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr 2024-03-29T10:35:07.716 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 10:34:08 AM UTC. 2024-03-29T10:35:07.816 INFO:teuthology.orchestra.run.smithi179.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:35:07.868 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:35:07.869 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-29T10:35:07.870 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:35:07.949 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:35:08.509 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 10:34:08 AM UTC. 2024-03-29T10:35:08.662 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:35:08.663 INFO:teuthology.orchestra.run.smithi179.stdout:====================================================================================== 2024-03-29T10:35:08.663 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-29T10:35:08.663 INFO:teuthology.orchestra.run.smithi179.stdout:====================================================================================== 2024-03-29T10:35:08.663 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T10:35:08.663 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:35:08.663 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-29T10:35:08.663 INFO:teuthology.orchestra.run.smithi179.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:35:08.663 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:35:08.663 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:35:08.663 INFO:teuthology.orchestra.run.smithi179.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:35:08.663 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:35:08.663 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:35:08.664 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:35:08.664 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:35:08.664 INFO:teuthology.orchestra.run.smithi179.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:35:08.664 INFO:teuthology.orchestra.run.smithi179.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:35:08.664 INFO:teuthology.orchestra.run.smithi179.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:35:08.664 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:08.664 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T10:35:08.664 INFO:teuthology.orchestra.run.smithi179.stdout:====================================================================================== 2024-03-29T10:35:08.664 INFO:teuthology.orchestra.run.smithi179.stdout:Install 12 Packages 2024-03-29T10:35:08.664 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:08.665 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 6.9 M 2024-03-29T10:35:08.665 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 26 M 2024-03-29T10:35:08.665 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T10:35:09.213 INFO:teuthology.orchestra.run.smithi179.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 170 kB/s | 31 kB 00:00 2024-03-29T10:35:09.321 INFO:teuthology.orchestra.run.smithi179.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 52 kB/s | 15 kB 00:00 2024-03-29T10:35:09.388 INFO:teuthology.orchestra.run.smithi179.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-03-29T10:35:09.422 INFO:teuthology.orchestra.run.smithi179.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.3 MB/s | 238 kB 00:00 2024-03-29T10:35:09.651 INFO:teuthology.orchestra.run.smithi179.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.2 MB/s | 863 kB 00:00 2024-03-29T10:35:09.864 INFO:teuthology.orchestra.run.smithi179.stdout:(6/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 2.1 MB/s | 1.8 MB 00:00 2024-03-29T10:35:09.906 INFO:teuthology.orchestra.run.smithi179.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 76 kB/s | 19 kB 00:00 2024-03-29T10:35:09.931 INFO:teuthology.orchestra.run.smithi179.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-03-29T10:35:10.031 INFO:teuthology.orchestra.run.smithi179.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 403 kB/s | 31 kB 00:00 2024-03-29T10:35:10.107 INFO:teuthology.orchestra.run.smithi179.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 719 kB/s | 144 kB 00:00 2024-03-29T10:35:10.132 INFO:teuthology.orchestra.run.smithi179.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-03-29T10:35:10.465 INFO:teuthology.orchestra.run.smithi179.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.4 MB/s | 2.0 MB 00:00 2024-03-29T10:35:10.471 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:10.471 INFO:teuthology.orchestra.run.smithi179.stdout:Total 3.8 MB/s | 6.9 MB 00:01 2024-03-29T10:35:10.490 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T10:35:10.507 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T10:35:10.507 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T10:35:10.677 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T10:35:10.677 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T10:35:11.194 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T10:35:11.282 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:35:11.318 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:35:11.364 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:35:11.395 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:35:11.498 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:35:11.621 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:35:11.706 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:35:11.757 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:35:11.786 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:35:11.818 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:35:12.062 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:35:12.093 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:35:12.914 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:35:12.914 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:35:12.914 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:35:12.914 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:35:12.915 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:35:12.915 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:35:12.915 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:35:12.915 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:35:12.915 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:35:12.915 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:35:12.915 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:35:12.916 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:35:13.274 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:35:13.274 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:13.274 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T10:35:13.274 INFO:teuthology.orchestra.run.smithi179.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:35:13.274 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:35:13.274 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:35:13.275 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:35:13.275 INFO:teuthology.orchestra.run.smithi179.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:35:13.275 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:35:13.275 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:35:13.275 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:35:13.275 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:35:13.275 INFO:teuthology.orchestra.run.smithi179.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:35:13.275 INFO:teuthology.orchestra.run.smithi179.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:35:13.275 INFO:teuthology.orchestra.run.smithi179.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:35:13.275 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:13.275 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:35:13.528 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:35:14.083 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:06 ago on Fri 29 Mar 2024 10:34:08 AM UTC. 2024-03-29T10:35:14.251 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:35:14.252 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================================ 2024-03-29T10:35:14.252 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-29T10:35:14.252 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================================ 2024-03-29T10:35:14.252 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T10:35:14.252 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:35:14.252 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-29T10:35:14.252 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:35:14.252 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:35:14.252 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:35:14.252 INFO:teuthology.orchestra.run.smithi179.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:35:14.252 INFO:teuthology.orchestra.run.smithi179.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:35:14.253 INFO:teuthology.orchestra.run.smithi179.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:35:14.253 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:35:14.253 INFO:teuthology.orchestra.run.smithi179.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:35:14.253 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:35:14.253 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:35:14.253 INFO:teuthology.orchestra.run.smithi179.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:35:14.253 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:14.253 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T10:35:14.253 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================================ 2024-03-29T10:35:14.253 INFO:teuthology.orchestra.run.smithi179.stdout:Install 12 Packages 2024-03-29T10:35:14.253 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:14.254 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 37 M 2024-03-29T10:35:14.254 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 209 M 2024-03-29T10:35:14.254 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T10:35:14.547 INFO:teuthology.orchestra.run.smithi179.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 188 kB/s | 32 kB 00:00 2024-03-29T10:35:14.613 INFO:teuthology.orchestra.run.smithi179.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 261 kB/s | 17 kB 00:00 2024-03-29T10:35:14.672 INFO:teuthology.orchestra.run.smithi179.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 585 kB/s | 34 kB 00:00 2024-03-29T10:35:14.789 INFO:teuthology.orchestra.run.smithi179.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.3 MB/s | 3.0 MB 00:00 2024-03-29T10:35:15.089 INFO:teuthology.orchestra.run.smithi179.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 11 MB/s | 4.7 MB 00:00 2024-03-29T10:35:15.156 INFO:teuthology.orchestra.run.smithi179.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 4.8 MB 00:00 2024-03-29T10:35:15.182 INFO:teuthology.orchestra.run.smithi179.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T10:35:15.632 INFO:teuthology.orchestra.run.smithi179.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-03-29T10:35:16.515 INFO:teuthology.orchestra.run.smithi179.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-14 3.5 MB/s | 7.4 MB 00:02 2024-03-29T10:35:16.691 INFO:teuthology.orchestra.run.smithi179.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 189 kB/s | 195 kB 00:01 2024-03-29T10:35:16.791 INFO:teuthology.orchestra.run.smithi179.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 893 kB/s | 245 kB 00:00 2024-03-29T10:35:17.216 INFO:teuthology.orchestra.run.smithi179.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 395 kB/s | 803 kB 00:02 2024-03-29T10:35:17.220 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:17.220 INFO:teuthology.orchestra.run.smithi179.stdout:Total 12 MB/s | 37 MB 00:02 2024-03-29T10:35:17.263 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T10:35:17.289 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T10:35:17.290 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T10:35:17.668 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T10:35:17.668 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T10:35:18.414 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T10:35:18.480 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:35:18.504 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:35:18.596 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:35:18.631 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:35:18.809 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:35:18.839 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:35:18.915 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:35:19.562 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:35:19.676 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:35:20.901 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:35:21.935 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:35:21.963 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:35:24.524 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:35:24.524 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:35:24.524 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:35:24.524 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:35:24.524 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:35:24.525 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:35:24.525 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:35:24.525 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:35:24.525 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:35:24.525 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:35:24.525 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:35:24.525 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:35:25.039 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:35:25.039 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:25.039 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T10:35:25.039 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:35:25.039 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:35:25.039 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:35:25.039 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:35:25.040 INFO:teuthology.orchestra.run.smithi179.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:35:25.040 INFO:teuthology.orchestra.run.smithi179.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:35:25.040 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:35:25.040 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:35:25.040 INFO:teuthology.orchestra.run.smithi179.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:35:25.040 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:35:25.040 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:35:25.040 INFO:teuthology.orchestra.run.smithi179.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:35:25.040 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:25.040 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:35:25.440 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:35:26.015 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:18 ago on Fri 29 Mar 2024 10:34:08 AM UTC. 2024-03-29T10:35:26.169 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:35:26.170 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-29T10:35:26.170 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-29T10:35:26.170 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-29T10:35:26.171 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T10:35:26.171 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:35:26.171 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-29T10:35:26.171 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:35:26.171 INFO:teuthology.orchestra.run.smithi179.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:35:26.171 INFO:teuthology.orchestra.run.smithi179.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:35:26.171 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:35:26.171 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:35:26.171 INFO:teuthology.orchestra.run.smithi179.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:35:26.171 INFO:teuthology.orchestra.run.smithi179.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:35:26.171 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:35:26.171 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:35:26.171 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:35:26.171 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:35:26.172 INFO:teuthology.orchestra.run.smithi179.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:35:26.172 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:26.172 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T10:35:26.172 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-29T10:35:26.172 INFO:teuthology.orchestra.run.smithi179.stdout:Install 13 Packages 2024-03-29T10:35:26.172 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:26.172 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 2.0 M 2024-03-29T10:35:26.172 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 26 M 2024-03-29T10:35:26.173 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T10:35:26.747 INFO:teuthology.orchestra.run.smithi179.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 148 kB/s | 24 kB 00:00 2024-03-29T10:35:26.781 INFO:teuthology.orchestra.run.smithi179.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 91 kB/s | 18 kB 00:00 2024-03-29T10:35:26.906 INFO:teuthology.orchestra.run.smithi179.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.1 MB/s | 174 kB 00:00 2024-03-29T10:35:26.932 INFO:teuthology.orchestra.run.smithi179.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 880 kB/s | 132 kB 00:00 2024-03-29T10:35:26.982 INFO:teuthology.orchestra.run.smithi179.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2024-03-29T10:35:27.007 INFO:teuthology.orchestra.run.smithi179.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 569 kB/s | 43 kB 00:00 2024-03-29T10:35:27.033 INFO:teuthology.orchestra.run.smithi179.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 116 kB/s | 52 kB 00:00 2024-03-29T10:35:27.116 INFO:teuthology.orchestra.run.smithi179.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 138 kB/s | 15 kB 00:00 2024-03-29T10:35:27.133 INFO:teuthology.orchestra.run.smithi179.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 213 kB/s | 32 kB 00:00 2024-03-29T10:35:27.300 INFO:teuthology.orchestra.run.smithi179.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 354 kB/s | 59 kB 00:00 2024-03-29T10:35:27.325 INFO:teuthology.orchestra.run.smithi179.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 693 kB/s | 203 kB 00:00 2024-03-29T10:35:27.492 INFO:teuthology.orchestra.run.smithi179.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 467 kB/s | 90 kB 00:00 2024-03-29T10:35:27.826 INFO:teuthology.orchestra.run.smithi179.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.4 MB/s | 1.0 MB 00:00 2024-03-29T10:35:27.829 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:27.829 INFO:teuthology.orchestra.run.smithi179.stdout:Total 1.2 MB/s | 2.0 MB 00:01 2024-03-29T10:35:27.829 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T10:35:27.844 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T10:35:27.844 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T10:35:27.941 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T10:35:27.958 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T10:35:28.473 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T10:35:28.638 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:35:28.702 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:35:28.759 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:35:28.795 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:35:28.836 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:35:28.938 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:35:29.065 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:35:29.120 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:35:29.838 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:35:29.893 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:35:29.926 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:35:29.975 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:35:29.997 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:35:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:35:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:35:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:35:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:35:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:35:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:35:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:35:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:35:30.828 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:35:30.829 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:35:30.829 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:35:30.829 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:35:30.829 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:35:31.197 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:35:31.197 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:31.198 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T10:35:31.198 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:35:31.198 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:35:31.198 INFO:teuthology.orchestra.run.smithi179.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:35:31.198 INFO:teuthology.orchestra.run.smithi179.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:35:31.198 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:35:31.198 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:35:31.198 INFO:teuthology.orchestra.run.smithi179.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:35:31.198 INFO:teuthology.orchestra.run.smithi179.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:35:31.198 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:35:31.198 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:35:31.199 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:35:31.199 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:35:31.199 INFO:teuthology.orchestra.run.smithi179.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:35:31.199 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:31.199 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:35:31.449 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:35:32.021 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:24 ago on Fri 29 Mar 2024 10:34:08 AM UTC. 2024-03-29T10:35:32.172 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:35:32.173 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-29T10:35:32.173 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-29T10:35:32.173 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-29T10:35:32.173 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T10:35:32.173 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:35:32.173 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-29T10:35:32.173 INFO:teuthology.orchestra.run.smithi179.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:35:32.173 INFO:teuthology.orchestra.run.smithi179.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:35:32.173 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:35:32.174 INFO:teuthology.orchestra.run.smithi179.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:35:32.174 INFO:teuthology.orchestra.run.smithi179.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:35:32.174 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:32.174 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T10:35:32.174 INFO:teuthology.orchestra.run.smithi179.stdout:======================================================================================== 2024-03-29T10:35:32.174 INFO:teuthology.orchestra.run.smithi179.stdout:Install 6 Packages 2024-03-29T10:35:32.174 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:32.175 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 6.8 M 2024-03-29T10:35:32.175 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 33 M 2024-03-29T10:35:32.175 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T10:35:32.413 INFO:teuthology.orchestra.run.smithi179.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.9 MB/s | 248 kB 00:00 2024-03-29T10:35:32.621 INFO:teuthology.orchestra.run.smithi179.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 413 kB/s | 138 kB 00:00 2024-03-29T10:35:32.797 INFO:teuthology.orchestra.run.smithi179.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 330 kB/s | 58 kB 00:00 2024-03-29T10:35:32.855 INFO:teuthology.orchestra.run.smithi179.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2024-03-29T10:35:32.872 INFO:teuthology.orchestra.run.smithi179.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 683 kB/s | 51 kB 00:00 2024-03-29T10:35:33.297 INFO:teuthology.orchestra.run.smithi179.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.7 MB/s | 5.8 MB 00:01 2024-03-29T10:35:33.300 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:33.300 INFO:teuthology.orchestra.run.smithi179.stdout:Total 6.0 MB/s | 6.8 MB 00:01 2024-03-29T10:35:33.313 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T10:35:33.324 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T10:35:33.324 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T10:35:33.416 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T10:35:33.416 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T10:35:33.722 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T10:35:33.881 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:35:33.932 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:35:34.446 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:35:34.535 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:35:34.600 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:35:34.628 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:35:35.476 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:35:35.477 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:35:35.477 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:35:35.477 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:35:35.477 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:35:35.477 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:35:35.793 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:35:35.793 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:35.793 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T10:35:35.793 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:35:35.793 INFO:teuthology.orchestra.run.smithi179.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:35:35.793 INFO:teuthology.orchestra.run.smithi179.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:35:35.793 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:35:35.793 INFO:teuthology.orchestra.run.smithi179.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:35:35.793 INFO:teuthology.orchestra.run.smithi179.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:35:35.794 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:35.794 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:35:36.058 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-fuse 2024-03-29T10:35:36.615 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:28 ago on Fri 29 Mar 2024 10:34:08 AM UTC. 2024-03-29T10:35:36.765 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:35:36.765 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:35:36.765 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-29T10:35:36.765 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:35:36.765 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T10:35:36.765 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:35:36.765 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-29T10:35:36.766 INFO:teuthology.orchestra.run.smithi179.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:35:36.766 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:36.766 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T10:35:36.766 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:35:36.766 INFO:teuthology.orchestra.run.smithi179.stdout:Install 2 Packages 2024-03-29T10:35:36.766 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:36.766 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 964 k 2024-03-29T10:35:36.767 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 2.8 M 2024-03-29T10:35:36.767 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T10:35:36.898 INFO:teuthology.orchestra.run.smithi179.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 597 kB/s | 79 kB 00:00 2024-03-29T10:35:37.499 INFO:teuthology.orchestra.run.smithi179.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 885 kB 00:00 2024-03-29T10:35:37.500 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:37.501 INFO:teuthology.orchestra.run.smithi179.stdout:Total 1.3 MB/s | 964 kB 00:00 2024-03-29T10:35:37.501 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T10:35:37.522 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T10:35:37.523 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T10:35:37.599 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T10:35:37.599 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T10:35:37.782 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T10:35:37.926 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:35:37.990 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:35:38.761 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:35:38.761 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:35:39.078 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:35:39.078 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:39.078 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T10:35:39.078 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:35:39.078 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:39.078 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:35:39.386 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-volume 2024-03-29T10:35:39.971 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:31 ago on Fri 29 Mar 2024 10:34:08 AM UTC. 2024-03-29T10:35:40.125 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:35:40.125 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:35:40.125 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-29T10:35:40.125 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:35:40.126 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T10:35:40.126 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:35:40.126 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-29T10:35:40.126 INFO:teuthology.orchestra.run.smithi179.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:35:40.126 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:40.126 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T10:35:40.126 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:35:40.126 INFO:teuthology.orchestra.run.smithi179.stdout:Install 2 Packages 2024-03-29T10:35:40.126 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:40.127 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 332 k 2024-03-29T10:35:40.127 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 1.5 M 2024-03-29T10:35:40.127 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T10:35:40.303 INFO:teuthology.orchestra.run.smithi179.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 390 kB/s | 69 kB 00:00 2024-03-29T10:35:40.603 INFO:teuthology.orchestra.run.smithi179.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 554 kB/s | 264 kB 00:00 2024-03-29T10:35:40.604 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:40.605 INFO:teuthology.orchestra.run.smithi179.stdout:Total 695 kB/s | 332 kB 00:00 2024-03-29T10:35:40.605 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T10:35:40.614 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T10:35:40.614 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T10:35:40.652 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T10:35:40.653 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T10:35:40.799 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T10:35:40.971 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:35:41.006 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:35:41.022 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:35:41.022 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:35:41.022 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:35:41.022 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:41.762 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:35:42.128 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:35:42.128 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:42.128 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T10:35:42.128 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:35:42.129 INFO:teuthology.orchestra.run.smithi179.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:35:42.129 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:42.129 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:35:42.494 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install librados-devel 2024-03-29T10:35:43.077 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 10:34:08 AM UTC. 2024-03-29T10:35:43.229 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:35:43.230 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:35:43.230 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-29T10:35:43.230 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:35:43.230 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T10:35:43.230 INFO:teuthology.orchestra.run.smithi179.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:35:43.230 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:43.230 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T10:35:43.230 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:35:43.230 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-29T10:35:43.230 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:43.230 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 127 k 2024-03-29T10:35:43.231 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 456 k 2024-03-29T10:35:43.231 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T10:35:43.568 INFO:teuthology.orchestra.run.smithi179.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 376 kB/s | 127 kB 00:00 2024-03-29T10:35:43.568 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:43.568 INFO:teuthology.orchestra.run.smithi179.stdout:Total 374 kB/s | 127 kB 00:00 2024-03-29T10:35:43.569 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T10:35:43.574 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T10:35:43.574 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T10:35:43.630 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T10:35:43.630 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T10:35:43.777 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T10:35:43.892 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:35:44.321 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:35:44.635 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:35:44.635 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:44.635 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T10:35:44.635 INFO:teuthology.orchestra.run.smithi179.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:35:44.635 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:44.635 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:35:44.867 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install libcephfs2 2024-03-29T10:35:45.450 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 10:34:08 AM UTC. 2024-03-29T10:35:45.553 INFO:teuthology.orchestra.run.smithi179.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:35:45.609 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:35:45.610 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-29T10:35:45.610 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:35:45.691 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install libcephfs-devel 2024-03-29T10:35:46.260 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 10:34:08 AM UTC. 2024-03-29T10:35:46.413 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:35:46.414 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:35:46.414 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-29T10:35:46.414 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:35:46.414 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T10:35:46.414 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:35:46.414 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:46.414 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T10:35:46.414 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:35:46.414 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-29T10:35:46.414 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:46.415 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 31 k 2024-03-29T10:35:46.415 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 140 k 2024-03-29T10:35:46.415 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T10:35:46.737 INFO:teuthology.orchestra.run.smithi179.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 96 kB/s | 31 kB 00:00 2024-03-29T10:35:46.738 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:46.738 INFO:teuthology.orchestra.run.smithi179.stdout:Total 96 kB/s | 31 kB 00:00 2024-03-29T10:35:46.738 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T10:35:46.742 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T10:35:46.742 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T10:35:46.770 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T10:35:46.770 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T10:35:46.907 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T10:35:47.002 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:35:47.386 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:35:47.656 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:35:47.657 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:47.657 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T10:35:47.657 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:35:47.657 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:47.657 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:35:47.861 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install librados2 2024-03-29T10:35:48.425 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 10:34:08 AM UTC. 2024-03-29T10:35:48.526 INFO:teuthology.orchestra.run.smithi179.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:35:48.578 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:35:48.579 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-29T10:35:48.579 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:35:48.633 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install librbd1 2024-03-29T10:35:49.194 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:34:08 AM UTC. 2024-03-29T10:35:49.289 INFO:teuthology.orchestra.run.smithi179.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:35:49.341 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:35:49.342 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-29T10:35:49.342 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:35:49.386 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-rados 2024-03-29T10:35:49.956 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:34:08 AM UTC. 2024-03-29T10:35:50.055 INFO:teuthology.orchestra.run.smithi179.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:35:50.106 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:35:50.107 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-29T10:35:50.108 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:35:50.159 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-rgw 2024-03-29T10:35:50.722 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:34:08 AM UTC. 2024-03-29T10:35:50.821 INFO:teuthology.orchestra.run.smithi179.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:35:50.873 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:35:50.874 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-29T10:35:50.874 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:35:50.927 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-cephfs 2024-03-29T10:35:51.497 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:34:08 AM UTC. 2024-03-29T10:35:51.595 INFO:teuthology.orchestra.run.smithi179.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:35:51.647 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:35:51.648 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-29T10:35:51.648 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:35:51.708 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install python3-rbd 2024-03-29T10:35:52.288 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:34:08 AM UTC. 2024-03-29T10:35:52.393 INFO:teuthology.orchestra.run.smithi179.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:35:52.449 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:35:52.450 INFO:teuthology.orchestra.run.smithi179.stdout:Nothing to do. 2024-03-29T10:35:52.450 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:35:52.504 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install rbd-fuse 2024-03-29T10:35:53.095 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 10:34:08 AM UTC. 2024-03-29T10:35:53.250 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:35:53.251 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:35:53.251 INFO:teuthology.orchestra.run.smithi179.stdout: Package Architecture Version Repository Size 2024-03-29T10:35:53.251 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:35:53.251 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T10:35:53.251 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:35:53.251 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:53.251 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T10:35:53.251 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:35:53.252 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-29T10:35:53.252 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:53.252 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 85 k 2024-03-29T10:35:53.252 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 231 k 2024-03-29T10:35:53.252 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T10:35:53.544 INFO:teuthology.orchestra.run.smithi179.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 292 kB/s | 85 kB 00:00 2024-03-29T10:35:53.545 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:53.545 INFO:teuthology.orchestra.run.smithi179.stdout:Total 290 kB/s | 85 kB 00:00 2024-03-29T10:35:53.545 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T10:35:53.552 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T10:35:53.552 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T10:35:53.608 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T10:35:53.608 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T10:35:53.750 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T10:35:53.856 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:35:54.261 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:35:54.556 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:35:54.556 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:54.557 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T10:35:54.557 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:35:54.557 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:54.557 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:35:54.819 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install rbd-mirror 2024-03-29T10:35:55.399 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 10:34:08 AM UTC. 2024-03-29T10:35:55.556 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:35:55.557 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:35:55.557 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repo Size 2024-03-29T10:35:55.557 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:35:55.557 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T10:35:55.557 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:35:55.557 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:55.557 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T10:35:55.557 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:35:55.557 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-29T10:35:55.558 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:35:55.558 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 3.1 M 2024-03-29T10:35:55.558 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 13 M 2024-03-29T10:35:55.558 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T10:35:58.836 INFO:teuthology.orchestra.run.smithi179.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 965 kB/s | 3.1 MB 00:03 2024-03-29T10:35:58.836 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:35:58.837 INFO:teuthology.orchestra.run.smithi179.stdout:Total 964 kB/s | 3.1 MB 00:03 2024-03-29T10:35:58.837 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T10:35:58.848 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T10:35:58.848 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T10:35:58.925 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T10:35:58.925 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T10:35:59.360 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T10:35:59.420 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:35:59.445 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:35:59.445 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:35:59.445 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:35:59.445 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:35:59.445 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:35:59.445 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:36:00.547 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:36:00.547 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:36:00.548 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T10:36:00.548 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:00.548 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:36:00.548 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:36:00.801 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install rbd-nbd 2024-03-29T10:36:01.376 INFO:teuthology.orchestra.run.smithi179.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 10:34:08 AM UTC. 2024-03-29T10:36:01.527 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-29T10:36:01.527 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:36:01.528 INFO:teuthology.orchestra.run.smithi179.stdout: Package Architecture Version Repository Size 2024-03-29T10:36:01.528 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:36:01.528 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-29T10:36:01.528 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:36:01.528 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:36:01.528 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-29T10:36:01.528 INFO:teuthology.orchestra.run.smithi179.stdout:================================================================================ 2024-03-29T10:36:01.528 INFO:teuthology.orchestra.run.smithi179.stdout:Install 1 Package 2024-03-29T10:36:01.528 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:36:01.528 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 171 k 2024-03-29T10:36:01.528 INFO:teuthology.orchestra.run.smithi179.stdout:Installed size: 494 k 2024-03-29T10:36:01.528 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-29T10:36:01.764 INFO:teuthology.orchestra.run.smithi179.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 729 kB/s | 171 kB 00:00 2024-03-29T10:36:01.764 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:36:01.765 INFO:teuthology.orchestra.run.smithi179.stdout:Total 725 kB/s | 171 kB 00:00 2024-03-29T10:36:01.765 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-29T10:36:01.772 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-29T10:36:01.772 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-29T10:36:01.828 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-29T10:36:01.828 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-29T10:36:01.974 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-29T10:36:02.095 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:36:02.560 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:36:02.838 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:36:02.838 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:36:02.838 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-29T10:36:02.838 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:36:02.839 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:36:02.839 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-29T10:36:03.156 DEBUG:teuthology.parallel:result is None 2024-03-29T10:36:44.176 INFO:teuthology.orchestra.run.smithi057.stdout:Extra Packages for Enterprise Linux 160 kB/s | 21 MB 02:12 2024-03-29T10:36:49.283 INFO:teuthology.orchestra.run.smithi057.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-29T10:36:51.992 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:36:52.007 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:36:52.007 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2024-03-29T10:36:52.007 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:36:52.007 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-03-29T10:36:52.007 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:36:52.007 INFO:teuthology.orchestra.run.smithi057.stdout:Upgrading: 2024-03-29T10:36:52.007 INFO:teuthology.orchestra.run.smithi057.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:36:52.007 INFO:teuthology.orchestra.run.smithi057.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:36:52.007 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2024-03-29T10:36:52.007 INFO:teuthology.orchestra.run.smithi057.stdout: boost-program-options 2024-03-29T10:36:52.007 INFO:teuthology.orchestra.run.smithi057.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:36:52.007 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:36:52.008 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:36:52.008 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:36:52.008 INFO:teuthology.orchestra.run.smithi057.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:36:52.008 INFO:teuthology.orchestra.run.smithi057.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:36:52.008 INFO:teuthology.orchestra.run.smithi057.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:36:52.008 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:36:52.008 INFO:teuthology.orchestra.run.smithi057.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:36:52.008 INFO:teuthology.orchestra.run.smithi057.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:36:52.008 INFO:teuthology.orchestra.run.smithi057.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:36:52.008 INFO:teuthology.orchestra.run.smithi057.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:36:52.008 INFO:teuthology.orchestra.run.smithi057.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:36:52.008 INFO:teuthology.orchestra.run.smithi057.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:36:52.008 INFO:teuthology.orchestra.run.smithi057.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:36:52.009 INFO:teuthology.orchestra.run.smithi057.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:36:52.009 INFO:teuthology.orchestra.run.smithi057.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:36:52.009 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-argparse 2024-03-29T10:36:52.009 INFO:teuthology.orchestra.run.smithi057.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:36:52.009 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:36:52.009 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:36:52.009 INFO:teuthology.orchestra.run.smithi057.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:36:52.009 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:36:52.009 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:36:52.009 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:36:52.009 INFO:teuthology.orchestra.run.smithi057.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:36:52.009 INFO:teuthology.orchestra.run.smithi057.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:36:52.009 INFO:teuthology.orchestra.run.smithi057.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:36:52.009 INFO:teuthology.orchestra.run.smithi057.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:36:52.009 INFO:teuthology.orchestra.run.smithi057.stdout:Installing weak dependencies: 2024-03-29T10:36:52.010 INFO:teuthology.orchestra.run.smithi057.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:36:52.010 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:36:52.010 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-03-29T10:36:52.010 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:36:52.010 INFO:teuthology.orchestra.run.smithi057.stdout:Install 30 Packages 2024-03-29T10:36:52.010 INFO:teuthology.orchestra.run.smithi057.stdout:Upgrade 2 Packages 2024-03-29T10:36:52.010 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:36:52.010 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 58 M 2024-03-29T10:36:52.011 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-03-29T10:36:54.526 INFO:teuthology.orchestra.run.smithi057.stdout:(1/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 8.2 MB/s | 20 MB 00:02 2024-03-29T10:36:54.626 INFO:teuthology.orchestra.run.smithi057.stdout:(2/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 2.1 MB/s | 5.3 MB 00:02 2024-03-29T10:36:54.651 INFO:teuthology.orchestra.run.smithi057.stdout:(3/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 201 kB/s | 25 kB 00:00 2024-03-29T10:36:54.743 INFO:teuthology.orchestra.run.smithi057.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 5.3 MB/s | 498 kB 00:00 2024-03-29T10:36:54.877 INFO:teuthology.orchestra.run.smithi057.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 3.5 MB/s | 9.7 MB 00:02 2024-03-29T10:36:54.961 INFO:teuthology.orchestra.run.smithi057.stdout:(6/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 2.2 MB/s | 742 kB 00:00 2024-03-29T10:36:54.994 INFO:teuthology.orchestra.run.smithi057.stdout:(7/32): python3-ceph-argparse-19.0.0-1439.g5dcb 385 kB/s | 45 kB 00:00 2024-03-29T10:36:55.061 INFO:teuthology.orchestra.run.smithi057.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.3 MB/s | 131 kB 00:00 2024-03-29T10:36:55.087 INFO:teuthology.orchestra.run.smithi057.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.7 MB/s | 162 kB 00:00 2024-03-29T10:36:55.254 INFO:teuthology.orchestra.run.smithi057.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 298 kB 00:00 2024-03-29T10:36:55.279 INFO:teuthology.orchestra.run.smithi057.stdout:(11/32): python3-rados-19.0.0-1439.g5dcb190d.el 1.4 MB/s | 320 kB 00:00 2024-03-29T10:36:55.346 INFO:teuthology.orchestra.run.smithi057.stdout:(12/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 8.5 MB/s | 5.1 MB 00:00 2024-03-29T10:36:55.388 INFO:teuthology.orchestra.run.smithi057.stdout:(13/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 742 kB/s | 99 kB 00:00 2024-03-29T10:36:55.455 INFO:teuthology.orchestra.run.smithi057.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 603 kB/s | 106 kB 00:00 2024-03-29T10:36:55.472 INFO:teuthology.orchestra.run.smithi057.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 787 kB/s | 44 kB 00:00 2024-03-29T10:36:55.505 INFO:teuthology.orchestra.run.smithi057.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2024-03-29T10:36:55.606 INFO:teuthology.orchestra.run.smithi057.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 661 kB 00:00 2024-03-29T10:36:55.631 INFO:teuthology.orchestra.run.smithi057.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 282 kB 00:00 2024-03-29T10:36:55.656 INFO:teuthology.orchestra.run.smithi057.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 274 kB/s | 41 kB 00:00 2024-03-29T10:36:55.681 INFO:teuthology.orchestra.run.smithi057.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 220 kB 00:00 2024-03-29T10:36:55.706 INFO:teuthology.orchestra.run.smithi057.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 464 kB/s | 35 kB 00:00 2024-03-29T10:36:55.732 INFO:teuthology.orchestra.run.smithi057.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 925 kB/s | 46 kB 00:00 2024-03-29T10:36:55.866 INFO:teuthology.orchestra.run.smithi057.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 195 kB/s | 26 kB 00:00 2024-03-29T10:36:55.958 INFO:teuthology.orchestra.run.smithi057.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-03-29T10:36:55.983 INFO:teuthology.orchestra.run.smithi057.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 415 kB/s | 49 kB 00:00 2024-03-29T10:36:56.016 INFO:teuthology.orchestra.run.smithi057.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-29T10:36:56.092 INFO:teuthology.orchestra.run.smithi057.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-29T10:36:56.167 INFO:teuthology.orchestra.run.smithi057.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 9.6 MB/s | 4.4 MB 00:00 2024-03-29T10:36:56.251 INFO:teuthology.orchestra.run.smithi057.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.1 MB/s | 838 kB 00:00 2024-03-29T10:36:56.284 INFO:teuthology.orchestra.run.smithi057.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 8.2 MB/s | 1.6 MB 00:00 2024-03-29T10:36:56.476 INFO:teuthology.orchestra.run.smithi057.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 14 MB/s | 3.1 MB 00:00 2024-03-29T10:36:57.260 INFO:teuthology.orchestra.run.smithi057.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 3.1 MB/s | 3.4 MB 00:01 2024-03-29T10:36:57.264 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:36:57.264 INFO:teuthology.orchestra.run.smithi057.stdout:Total 11 MB/s | 58 MB 00:05 2024-03-29T10:36:57.265 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-03-29T10:36:57.307 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-03-29T10:36:57.307 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-03-29T10:36:57.805 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-03-29T10:36:57.805 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-03-29T10:36:59.443 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-03-29T10:36:59.522 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:36:59.556 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:36:59.582 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:36:59.613 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:36:59.629 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:36:59.821 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:36:59.877 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:37:00.228 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:37:00.284 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:37:00.375 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:37:00.466 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:37:00.494 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:37:00.556 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:37:00.585 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:37:00.629 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:37:00.699 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:37:00.721 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:37:00.785 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:37:00.820 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:37:00.930 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:37:00.962 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:37:01.003 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:37:01.035 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:37:01.065 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:37:01.106 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:37:01.440 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:37:01.469 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:37:01.542 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:37:01.574 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:37:01.613 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:37:02.088 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:37:02.122 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:37:02.193 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:37:02.263 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:37:02.291 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:37:04.511 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:37:04.554 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:37:05.131 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:37:05.166 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:37:05.244 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:37:05.244 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:37:05.244 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:37:05.244 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:05.276 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:37:15.396 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:37:15.396 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /sys 2024-03-29T10:37:15.396 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /proc 2024-03-29T10:37:15.396 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /mnt 2024-03-29T10:37:15.396 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /var/tmp 2024-03-29T10:37:15.396 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /home 2024-03-29T10:37:15.396 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /root 2024-03-29T10:37:15.397 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /tmp 2024-03-29T10:37:15.397 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:16.331 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:37:16.356 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:37:16.356 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:37:16.356 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:37:16.356 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:37:16.356 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:37:16.356 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:16.358 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:37:16.442 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:37:16.442 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:37:17.579 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:37:17.579 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:37:17.579 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:37:17.580 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:37:17.580 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:37:17.580 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:37:17.580 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:37:17.580 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:37:17.580 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:37:17.580 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:37:17.580 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:37:17.580 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:37:17.580 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:37:17.580 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:37:17.580 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:37:17.580 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:37:17.581 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:37:17.581 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:37:17.581 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:37:17.581 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:37:17.581 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:37:17.581 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:37:17.581 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:37:17.581 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:37:17.581 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:37:17.581 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:37:17.582 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:37:17.582 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:37:17.583 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:37:17.583 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:37:17.583 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:37:17.583 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:37:17.583 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:37:17.583 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:37:18.685 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:37:18.685 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:18.685 INFO:teuthology.orchestra.run.smithi057.stdout:Upgraded: 2024-03-29T10:37:18.685 INFO:teuthology.orchestra.run.smithi057.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:18.685 INFO:teuthology.orchestra.run.smithi057.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:18.685 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-03-29T10:37:18.685 INFO:teuthology.orchestra.run.smithi057.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:37:18.685 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:18.685 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:18.686 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:18.686 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:18.686 INFO:teuthology.orchestra.run.smithi057.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:37:18.686 INFO:teuthology.orchestra.run.smithi057.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:37:18.686 INFO:teuthology.orchestra.run.smithi057.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:37:18.686 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:18.686 INFO:teuthology.orchestra.run.smithi057.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:37:18.686 INFO:teuthology.orchestra.run.smithi057.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:37:18.686 INFO:teuthology.orchestra.run.smithi057.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:37:18.686 INFO:teuthology.orchestra.run.smithi057.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:18.686 INFO:teuthology.orchestra.run.smithi057.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:37:18.686 INFO:teuthology.orchestra.run.smithi057.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:18.686 INFO:teuthology.orchestra.run.smithi057.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:37:18.686 INFO:teuthology.orchestra.run.smithi057.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:37:18.687 INFO:teuthology.orchestra.run.smithi057.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:37:18.687 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:18.687 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:18.687 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:18.687 INFO:teuthology.orchestra.run.smithi057.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:37:18.687 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:18.687 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:18.687 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:18.687 INFO:teuthology.orchestra.run.smithi057.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:37:18.687 INFO:teuthology.orchestra.run.smithi057.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:37:18.687 INFO:teuthology.orchestra.run.smithi057.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:37:18.687 INFO:teuthology.orchestra.run.smithi057.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:37:18.687 INFO:teuthology.orchestra.run.smithi057.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:37:18.687 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:18.687 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:37:19.020 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-test 2024-03-29T10:37:19.588 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:00:30 ago on Fri 29 Mar 2024 10:36:49 AM UTC. 2024-03-29T10:37:19.753 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:37:19.753 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:37:19.753 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2024-03-29T10:37:19.753 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:37:19.754 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-03-29T10:37:19.754 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:37:19.754 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2024-03-29T10:37:19.754 INFO:teuthology.orchestra.run.smithi057.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:37:19.754 INFO:teuthology.orchestra.run.smithi057.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:37:19.754 INFO:teuthology.orchestra.run.smithi057.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:37:19.754 INFO:teuthology.orchestra.run.smithi057.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:37:19.754 INFO:teuthology.orchestra.run.smithi057.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:37:19.754 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:19.754 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-03-29T10:37:19.754 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:37:19.755 INFO:teuthology.orchestra.run.smithi057.stdout:Install 6 Packages 2024-03-29T10:37:19.755 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:19.755 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 48 M 2024-03-29T10:37:19.755 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 204 M 2024-03-29T10:37:19.755 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-03-29T10:37:20.189 INFO:teuthology.orchestra.run.smithi057.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T10:37:20.272 INFO:teuthology.orchestra.run.smithi057.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.5 MB/s | 300 kB 00:00 2024-03-29T10:37:20.297 INFO:teuthology.orchestra.run.smithi057.stdout:(3/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 610 kB/s | 163 kB 00:00 2024-03-29T10:37:20.314 INFO:teuthology.orchestra.run.smithi057.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-29T10:37:20.448 INFO:teuthology.orchestra.run.smithi057.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T10:37:28.581 INFO:teuthology.orchestra.run.smithi057.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 5.5 MB/s | 47 MB 00:08 2024-03-29T10:37:28.583 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:28.584 INFO:teuthology.orchestra.run.smithi057.stdout:Total 5.5 MB/s | 48 MB 00:08 2024-03-29T10:37:28.598 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-03-29T10:37:28.622 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-03-29T10:37:28.622 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-03-29T10:37:29.022 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-03-29T10:37:29.023 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-03-29T10:37:29.630 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-03-29T10:37:29.691 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:37:29.726 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:37:29.759 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:37:29.805 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:37:29.834 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:37:34.559 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:37:34.651 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:37:36.949 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:37:36.949 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:37:36.949 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:37:36.949 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:37:36.949 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:37:36.949 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:37:37.358 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:37:37.358 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:37.358 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-03-29T10:37:37.358 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:37.358 INFO:teuthology.orchestra.run.smithi057.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:37:37.358 INFO:teuthology.orchestra.run.smithi057.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:37.358 INFO:teuthology.orchestra.run.smithi057.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:37:37.358 INFO:teuthology.orchestra.run.smithi057.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:37:37.359 INFO:teuthology.orchestra.run.smithi057.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:37:37.359 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:37.359 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:37:37.685 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph 2024-03-29T10:37:38.254 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:00:49 ago on Fri 29 Mar 2024 10:36:49 AM UTC. 2024-03-29T10:37:38.421 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:37:38.423 INFO:teuthology.orchestra.run.smithi057.stdout:========================================================================================= 2024-03-29T10:37:38.423 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2024-03-29T10:37:38.423 INFO:teuthology.orchestra.run.smithi057.stdout:========================================================================================= 2024-03-29T10:37:38.424 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-03-29T10:37:38.424 INFO:teuthology.orchestra.run.smithi057.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:37:38.424 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2024-03-29T10:37:38.424 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:37:38.424 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:37:38.424 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:37:38.424 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:37:38.425 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:37:38.425 INFO:teuthology.orchestra.run.smithi057.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:37:38.425 INFO:teuthology.orchestra.run.smithi057.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:37:38.425 INFO:teuthology.orchestra.run.smithi057.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:37:38.425 INFO:teuthology.orchestra.run.smithi057.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:37:38.425 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:37:38.425 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:37:38.425 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:37:38.426 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:37:38.426 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:37:38.426 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:37:38.426 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:37:38.426 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:37:38.426 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:37:38.426 INFO:teuthology.orchestra.run.smithi057.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:37:38.426 INFO:teuthology.orchestra.run.smithi057.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:37:38.426 INFO:teuthology.orchestra.run.smithi057.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:37:38.427 INFO:teuthology.orchestra.run.smithi057.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:37:38.427 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:37:38.427 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:37:38.427 INFO:teuthology.orchestra.run.smithi057.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:37:38.427 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:37:38.427 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:37:38.427 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:37:38.427 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:37:38.427 INFO:teuthology.orchestra.run.smithi057.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:37:38.427 INFO:teuthology.orchestra.run.smithi057.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:37:38.427 INFO:teuthology.orchestra.run.smithi057.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:37:38.428 INFO:teuthology.orchestra.run.smithi057.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:37:38.428 INFO:teuthology.orchestra.run.smithi057.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:37:38.428 INFO:teuthology.orchestra.run.smithi057.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:37:38.428 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:38.428 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-03-29T10:37:38.428 INFO:teuthology.orchestra.run.smithi057.stdout:========================================================================================= 2024-03-29T10:37:38.428 INFO:teuthology.orchestra.run.smithi057.stdout:Install 36 Packages 2024-03-29T10:37:38.428 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:38.429 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 30 M 2024-03-29T10:37:38.429 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 107 M 2024-03-29T10:37:38.429 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-03-29T10:37:38.974 INFO:teuthology.orchestra.run.smithi057.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 27 kB/s | 6.6 kB 00:00 2024-03-29T10:37:39.108 INFO:teuthology.orchestra.run.smithi057.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 4.0 MB/s | 1.5 MB 00:00 2024-03-29T10:37:39.191 INFO:teuthology.orchestra.run.smithi057.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 4.6 MB/s | 2.1 MB 00:00 2024-03-29T10:37:39.241 INFO:teuthology.orchestra.run.smithi057.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 4.8 MB/s | 247 kB 00:00 2024-03-29T10:37:39.350 INFO:teuthology.orchestra.run.smithi057.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-29T10:37:39.384 INFO:teuthology.orchestra.run.smithi057.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-03-29T10:37:39.417 INFO:teuthology.orchestra.run.smithi057.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 954 kB/s | 32 kB 00:00 2024-03-29T10:37:39.451 INFO:teuthology.orchestra.run.smithi057.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T10:37:39.484 INFO:teuthology.orchestra.run.smithi057.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T10:37:39.526 INFO:teuthology.orchestra.run.smithi057.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-29T10:37:39.585 INFO:teuthology.orchestra.run.smithi057.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 20 MB/s | 1.1 MB 00:00 2024-03-29T10:37:39.618 INFO:teuthology.orchestra.run.smithi057.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T10:37:39.652 INFO:teuthology.orchestra.run.smithi057.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T10:37:39.686 INFO:teuthology.orchestra.run.smithi057.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-29T10:37:39.886 INFO:teuthology.orchestra.run.smithi057.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 624 kB/s | 125 kB 00:00 2024-03-29T10:37:40.453 INFO:teuthology.orchestra.run.smithi057.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 39 kB/s | 22 kB 00:00 2024-03-29T10:37:40.728 INFO:teuthology.orchestra.run.smithi057.stdout:(17/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 2.7 MB/s | 4.7 MB 00:01 2024-03-29T10:37:40.812 INFO:teuthology.orchestra.run.smithi057.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 422 kB/s | 151 kB 00:00 2024-03-29T10:37:40.828 INFO:teuthology.orchestra.run.smithi057.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 434 kB/s | 43 kB 00:00 2024-03-29T10:37:40.912 INFO:teuthology.orchestra.run.smithi057.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-03-29T10:37:40.971 INFO:teuthology.orchestra.run.smithi057.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.5 MB/s | 358 kB 00:00 2024-03-29T10:37:40.987 INFO:teuthology.orchestra.run.smithi057.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 143 kB/s | 11 kB 00:00 2024-03-29T10:37:41.155 INFO:teuthology.orchestra.run.smithi057.stdout:(23/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 8.2 MB/s | 17 MB 00:02 2024-03-29T10:37:41.180 INFO:teuthology.orchestra.run.smithi057.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 85 kB/s | 18 kB 00:00 2024-03-29T10:37:41.205 INFO:teuthology.orchestra.run.smithi057.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 107 kB/s | 23 kB 00:00 2024-03-29T10:37:41.231 INFO:teuthology.orchestra.run.smithi057.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-03-29T10:37:41.273 INFO:teuthology.orchestra.run.smithi057.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 294 kB/s | 20 kB 00:00 2024-03-29T10:37:41.298 INFO:teuthology.orchestra.run.smithi057.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 503 kB/s | 46 kB 00:00 2024-03-29T10:37:41.323 INFO:teuthology.orchestra.run.smithi057.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 855 kB/s | 79 kB 00:00 2024-03-29T10:37:41.357 INFO:teuthology.orchestra.run.smithi057.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-29T10:37:41.390 INFO:teuthology.orchestra.run.smithi057.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 178 kB/s | 16 kB 00:00 2024-03-29T10:37:41.416 INFO:teuthology.orchestra.run.smithi057.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 978 kB/s | 90 kB 00:00 2024-03-29T10:37:41.441 INFO:teuthology.orchestra.run.smithi057.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 427 kB/s | 36 kB 00:00 2024-03-29T10:37:41.466 INFO:teuthology.orchestra.run.smithi057.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-29T10:37:41.508 INFO:teuthology.orchestra.run.smithi057.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-03-29T10:37:41.533 INFO:teuthology.orchestra.run.smithi057.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-03-29T10:37:41.539 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:41.539 INFO:teuthology.orchestra.run.smithi057.stdout:Total 9.5 MB/s | 30 MB 00:03 2024-03-29T10:37:41.563 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-03-29T10:37:41.608 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-03-29T10:37:41.608 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-03-29T10:37:42.022 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-03-29T10:37:42.023 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-03-29T10:37:43.404 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-03-29T10:37:43.477 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:37:43.511 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:37:43.538 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:37:43.612 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:37:43.653 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:37:43.686 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:37:43.781 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:37:43.867 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:37:43.914 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:37:43.997 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:37:44.049 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:37:44.086 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:37:44.136 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:37:44.191 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:37:44.382 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:37:44.432 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:37:44.470 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:37:44.505 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:37:44.534 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:37:44.613 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:37:44.653 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:37:44.686 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:37:44.727 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:37:44.762 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:37:44.796 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:37:44.889 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:37:44.927 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:37:45.010 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:37:45.150 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:37:45.274 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:37:45.442 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:37:45.481 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:37:45.507 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:37:45.507 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:37:45.507 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:37:45.507 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:37:45.507 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:37:45.508 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:47.015 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:37:47.041 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:37:47.042 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:37:47.042 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:37:47.042 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:37:47.042 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:37:47.042 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:47.499 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:37:47.523 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:37:47.523 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:37:47.523 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:37:47.523 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:37:47.523 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:37:47.523 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:47.741 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:37:47.765 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:37:47.765 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:37:47.766 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:37:47.766 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:37:47.766 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:37:47.766 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:47.834 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:37:49.583 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:37:49.583 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:37:49.583 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:37:49.583 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:37:49.583 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:37:49.583 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:37:49.583 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:37:49.583 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:37:49.583 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:37:49.584 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:37:49.584 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:37:49.584 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:37:49.584 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:37:49.584 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:37:49.584 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:37:49.584 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:37:49.584 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:37:49.584 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:37:49.584 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:37:49.584 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:37:49.584 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:37:49.584 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:37:49.584 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:37:49.585 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:37:49.585 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:37:49.585 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:37:49.585 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:37:49.585 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:37:49.585 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:37:49.585 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:37:49.585 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:37:49.585 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:37:49.585 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:37:49.585 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:37:49.585 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:37:49.585 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:37:50.387 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:37:50.387 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:50.387 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-03-29T10:37:50.387 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:50.387 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:50.388 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:50.388 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:50.388 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:50.388 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:50.388 INFO:teuthology.orchestra.run.smithi057.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:37:50.388 INFO:teuthology.orchestra.run.smithi057.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:37:50.388 INFO:teuthology.orchestra.run.smithi057.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:37:50.388 INFO:teuthology.orchestra.run.smithi057.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:37:50.388 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:37:50.388 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:37:50.388 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:37:50.388 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:37:50.389 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:37:50.389 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:37:50.389 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:37:50.389 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:37:50.389 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:37:50.389 INFO:teuthology.orchestra.run.smithi057.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:37:50.389 INFO:teuthology.orchestra.run.smithi057.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:37:50.389 INFO:teuthology.orchestra.run.smithi057.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:37:50.389 INFO:teuthology.orchestra.run.smithi057.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:37:50.389 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:37:50.389 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:37:50.389 INFO:teuthology.orchestra.run.smithi057.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:37:50.389 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:37:50.390 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:37:50.390 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:37:50.390 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:37:50.390 INFO:teuthology.orchestra.run.smithi057.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:37:50.390 INFO:teuthology.orchestra.run.smithi057.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:37:50.390 INFO:teuthology.orchestra.run.smithi057.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:37:50.390 INFO:teuthology.orchestra.run.smithi057.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:37:50.390 INFO:teuthology.orchestra.run.smithi057.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:37:50.390 INFO:teuthology.orchestra.run.smithi057.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:37:50.390 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:50.390 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:37:50.778 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-base 2024-03-29T10:37:51.362 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:02 ago on Fri 29 Mar 2024 10:36:49 AM UTC. 2024-03-29T10:37:51.463 INFO:teuthology.orchestra.run.smithi057.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:37:51.518 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:37:51.519 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2024-03-29T10:37:51.519 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:37:51.582 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install cephadm 2024-03-29T10:37:52.155 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:03 ago on Fri 29 Mar 2024 10:36:49 AM UTC. 2024-03-29T10:37:52.314 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:37:52.315 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:37:52.315 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2024-03-29T10:37:52.315 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:37:52.315 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-03-29T10:37:52.315 INFO:teuthology.orchestra.run.smithi057.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:37:52.315 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:52.316 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-03-29T10:37:52.316 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:37:52.316 INFO:teuthology.orchestra.run.smithi057.stdout:Install 1 Package 2024-03-29T10:37:52.316 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:52.316 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 747 k 2024-03-29T10:37:52.316 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 754 k 2024-03-29T10:37:52.316 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-03-29T10:37:52.612 INFO:teuthology.orchestra.run.smithi057.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.5 MB/s | 747 kB 00:00 2024-03-29T10:37:52.613 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:52.613 INFO:teuthology.orchestra.run.smithi057.stdout:Total 2.5 MB/s | 747 kB 00:00 2024-03-29T10:37:52.613 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-03-29T10:37:52.618 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-03-29T10:37:52.618 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-03-29T10:37:52.624 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-03-29T10:37:52.624 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-03-29T10:37:52.702 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-03-29T10:37:53.145 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:37:53.252 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:37:53.649 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:37:53.939 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:37:53.939 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:53.939 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-03-29T10:37:53.939 INFO:teuthology.orchestra.run.smithi057.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:37:53.939 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:53.939 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:37:54.142 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:37:54.714 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:05 ago on Fri 29 Mar 2024 10:36:49 AM UTC. 2024-03-29T10:37:54.871 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:37:54.871 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:37:54.872 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repo Size 2024-03-29T10:37:54.872 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:37:54.872 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-03-29T10:37:54.872 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:37:54.872 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:54.872 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-03-29T10:37:54.872 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:37:54.872 INFO:teuthology.orchestra.run.smithi057.stdout:Install 1 Package 2024-03-29T10:37:54.872 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:54.872 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 145 k 2024-03-29T10:37:54.872 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 436 k 2024-03-29T10:37:54.872 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-03-29T10:37:55.075 INFO:teuthology.orchestra.run.smithi057.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 717 kB/s | 145 kB 00:00 2024-03-29T10:37:55.076 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:55.076 INFO:teuthology.orchestra.run.smithi057.stdout:Total 712 kB/s | 145 kB 00:00 2024-03-29T10:37:55.076 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-03-29T10:37:55.084 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-03-29T10:37:55.084 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-03-29T10:37:55.139 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-03-29T10:37:55.139 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-03-29T10:37:55.277 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-03-29T10:37:55.335 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:37:55.359 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:37:55.360 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:37:55.360 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:37:55.360 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:56.321 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:37:56.322 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:56.322 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-03-29T10:37:56.322 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:37:56.322 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:56.322 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:37:56.543 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-mgr 2024-03-29T10:37:57.118 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:08 ago on Fri 29 Mar 2024 10:36:49 AM UTC. 2024-03-29T10:37:57.221 INFO:teuthology.orchestra.run.smithi057.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:37:57.274 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:37:57.275 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2024-03-29T10:37:57.275 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:37:57.343 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:37:57.916 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:08 ago on Fri 29 Mar 2024 10:36:49 AM UTC. 2024-03-29T10:37:58.072 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:37:58.073 INFO:teuthology.orchestra.run.smithi057.stdout:====================================================================================== 2024-03-29T10:37:58.073 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2024-03-29T10:37:58.073 INFO:teuthology.orchestra.run.smithi057.stdout:====================================================================================== 2024-03-29T10:37:58.074 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-03-29T10:37:58.074 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:37:58.074 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2024-03-29T10:37:58.074 INFO:teuthology.orchestra.run.smithi057.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:37:58.074 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:37:58.074 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:37:58.074 INFO:teuthology.orchestra.run.smithi057.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:37:58.074 INFO:teuthology.orchestra.run.smithi057.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:37:58.074 INFO:teuthology.orchestra.run.smithi057.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:37:58.074 INFO:teuthology.orchestra.run.smithi057.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:37:58.074 INFO:teuthology.orchestra.run.smithi057.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:37:58.074 INFO:teuthology.orchestra.run.smithi057.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:37:58.075 INFO:teuthology.orchestra.run.smithi057.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:37:58.075 INFO:teuthology.orchestra.run.smithi057.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:37:58.075 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:58.075 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-03-29T10:37:58.075 INFO:teuthology.orchestra.run.smithi057.stdout:====================================================================================== 2024-03-29T10:37:58.075 INFO:teuthology.orchestra.run.smithi057.stdout:Install 12 Packages 2024-03-29T10:37:58.075 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:37:58.076 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 6.9 M 2024-03-29T10:37:58.076 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 26 M 2024-03-29T10:37:58.076 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-03-29T10:37:58.411 INFO:teuthology.orchestra.run.smithi057.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 168 kB/s | 31 kB 00:00 2024-03-29T10:37:58.503 INFO:teuthology.orchestra.run.smithi057.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 55 kB/s | 15 kB 00:00 2024-03-29T10:37:58.562 INFO:teuthology.orchestra.run.smithi057.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-03-29T10:37:58.587 INFO:teuthology.orchestra.run.smithi057.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-29T10:37:58.687 INFO:teuthology.orchestra.run.smithi057.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 6.8 MB/s | 863 kB 00:00 2024-03-29T10:37:58.771 INFO:teuthology.orchestra.run.smithi057.stdout:(6/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.3 MB/s | 1.8 MB 00:00 2024-03-29T10:37:58.913 INFO:teuthology.orchestra.run.smithi057.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 86 kB/s | 19 kB 00:00 2024-03-29T10:37:59.296 INFO:teuthology.orchestra.run.smithi057.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 773 kB/s | 548 kB 00:00 2024-03-29T10:37:59.388 INFO:teuthology.orchestra.run.smithi057.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 304 kB/s | 144 kB 00:00 2024-03-29T10:37:59.688 INFO:teuthology.orchestra.run.smithi057.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 78 kB/s | 31 kB 00:00 2024-03-29T10:37:59.730 INFO:teuthology.orchestra.run.smithi057.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.1 MB/s | 2.0 MB 00:00 2024-03-29T10:37:59.764 INFO:teuthology.orchestra.run.smithi057.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 502 kB/s | 188 kB 00:00 2024-03-29T10:37:59.767 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:37:59.767 INFO:teuthology.orchestra.run.smithi057.stdout:Total 4.1 MB/s | 6.9 MB 00:01 2024-03-29T10:37:59.788 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-03-29T10:37:59.808 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-03-29T10:37:59.808 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-03-29T10:37:59.999 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-03-29T10:38:00.000 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-03-29T10:38:00.522 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-03-29T10:38:00.615 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:38:00.650 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:38:00.698 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:38:00.727 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:38:00.832 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:38:00.967 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:38:01.059 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:38:01.107 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:38:01.134 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:38:01.167 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:38:01.424 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:38:01.458 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:38:02.203 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:38:02.203 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:38:02.203 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:38:02.203 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:38:02.203 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:38:02.203 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:38:02.203 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:38:02.204 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:38:02.204 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:38:02.204 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:38:02.204 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:38:02.204 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:38:02.564 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:38:02.564 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:02.564 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-03-29T10:38:02.564 INFO:teuthology.orchestra.run.smithi057.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:38:02.564 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:38:02.564 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:38:02.564 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:38:02.564 INFO:teuthology.orchestra.run.smithi057.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:38:02.565 INFO:teuthology.orchestra.run.smithi057.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:38:02.565 INFO:teuthology.orchestra.run.smithi057.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:38:02.565 INFO:teuthology.orchestra.run.smithi057.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:38:02.565 INFO:teuthology.orchestra.run.smithi057.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:38:02.565 INFO:teuthology.orchestra.run.smithi057.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:38:02.565 INFO:teuthology.orchestra.run.smithi057.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:38:02.565 INFO:teuthology.orchestra.run.smithi057.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:38:02.565 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:02.565 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:38:02.819 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:38:03.389 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:14 ago on Fri 29 Mar 2024 10:36:49 AM UTC. 2024-03-29T10:38:03.546 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:38:03.547 INFO:teuthology.orchestra.run.smithi057.stdout:============================================================================================ 2024-03-29T10:38:03.547 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2024-03-29T10:38:03.547 INFO:teuthology.orchestra.run.smithi057.stdout:============================================================================================ 2024-03-29T10:38:03.547 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-03-29T10:38:03.547 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:38:03.547 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2024-03-29T10:38:03.548 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:38:03.548 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:38:03.548 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:38:03.548 INFO:teuthology.orchestra.run.smithi057.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:38:03.548 INFO:teuthology.orchestra.run.smithi057.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:38:03.548 INFO:teuthology.orchestra.run.smithi057.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:38:03.548 INFO:teuthology.orchestra.run.smithi057.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:38:03.548 INFO:teuthology.orchestra.run.smithi057.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:38:03.548 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:38:03.548 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:38:03.548 INFO:teuthology.orchestra.run.smithi057.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:38:03.548 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:03.548 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-03-29T10:38:03.549 INFO:teuthology.orchestra.run.smithi057.stdout:============================================================================================ 2024-03-29T10:38:03.549 INFO:teuthology.orchestra.run.smithi057.stdout:Install 12 Packages 2024-03-29T10:38:03.549 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:03.549 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 37 M 2024-03-29T10:38:03.549 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 209 M 2024-03-29T10:38:03.549 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-03-29T10:38:04.103 INFO:teuthology.orchestra.run.smithi057.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 238 kB/s | 32 kB 00:00 2024-03-29T10:38:04.136 INFO:teuthology.orchestra.run.smithi057.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 536 kB/s | 17 kB 00:00 2024-03-29T10:38:04.203 INFO:teuthology.orchestra.run.smithi057.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-03-29T10:38:04.228 INFO:teuthology.orchestra.run.smithi057.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 378 kB/s | 34 kB 00:00 2024-03-29T10:38:04.396 INFO:teuthology.orchestra.run.smithi057.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2024-03-29T10:38:04.462 INFO:teuthology.orchestra.run.smithi057.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-29T10:38:04.488 INFO:teuthology.orchestra.run.smithi057.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T10:38:04.830 INFO:teuthology.orchestra.run.smithi057.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-29T10:38:04.972 INFO:teuthology.orchestra.run.smithi057.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-29T10:38:04.997 INFO:teuthology.orchestra.run.smithi057.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-03-29T10:38:05.214 INFO:teuthology.orchestra.run.smithi057.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-03-29T10:38:06.172 INFO:teuthology.orchestra.run.smithi057.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 3.4 MB/s | 7.4 MB 00:02 2024-03-29T10:38:06.174 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:06.175 INFO:teuthology.orchestra.run.smithi057.stdout:Total 14 MB/s | 37 MB 00:02 2024-03-29T10:38:06.219 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-03-29T10:38:06.245 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-03-29T10:38:06.246 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-03-29T10:38:06.627 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-03-29T10:38:06.627 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-03-29T10:38:07.344 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-03-29T10:38:07.429 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:38:07.460 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:38:07.551 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:38:07.596 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:38:07.780 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:38:07.812 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:38:07.897 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:38:08.585 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:38:08.696 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:38:10.014 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:38:11.086 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:38:11.118 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:38:13.673 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:38:13.673 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:38:13.673 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:38:13.673 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:38:13.673 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:38:13.673 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:38:13.674 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:38:13.674 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:38:13.674 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:38:13.674 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:38:13.674 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:38:13.674 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:38:14.089 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:38:14.089 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:14.089 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-03-29T10:38:14.089 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:38:14.089 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:38:14.089 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:38:14.089 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:38:14.089 INFO:teuthology.orchestra.run.smithi057.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:38:14.089 INFO:teuthology.orchestra.run.smithi057.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:38:14.089 INFO:teuthology.orchestra.run.smithi057.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:38:14.089 INFO:teuthology.orchestra.run.smithi057.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:38:14.090 INFO:teuthology.orchestra.run.smithi057.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:38:14.090 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:38:14.090 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:38:14.090 INFO:teuthology.orchestra.run.smithi057.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:38:14.090 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:14.090 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:38:14.486 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:38:15.062 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:26 ago on Fri 29 Mar 2024 10:36:49 AM UTC. 2024-03-29T10:38:15.219 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:38:15.221 INFO:teuthology.orchestra.run.smithi057.stdout:======================================================================================== 2024-03-29T10:38:15.221 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2024-03-29T10:38:15.221 INFO:teuthology.orchestra.run.smithi057.stdout:======================================================================================== 2024-03-29T10:38:15.221 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-03-29T10:38:15.221 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:38:15.221 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2024-03-29T10:38:15.221 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:38:15.221 INFO:teuthology.orchestra.run.smithi057.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:38:15.221 INFO:teuthology.orchestra.run.smithi057.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:38:15.221 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:38:15.221 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:38:15.221 INFO:teuthology.orchestra.run.smithi057.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:38:15.222 INFO:teuthology.orchestra.run.smithi057.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:38:15.222 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:38:15.222 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:38:15.222 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:38:15.222 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:38:15.222 INFO:teuthology.orchestra.run.smithi057.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:38:15.222 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:15.222 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-03-29T10:38:15.222 INFO:teuthology.orchestra.run.smithi057.stdout:======================================================================================== 2024-03-29T10:38:15.222 INFO:teuthology.orchestra.run.smithi057.stdout:Install 13 Packages 2024-03-29T10:38:15.222 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:15.223 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 2.0 M 2024-03-29T10:38:15.223 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 26 M 2024-03-29T10:38:15.223 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-03-29T10:38:15.436 INFO:teuthology.orchestra.run.smithi057.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 180 kB/s | 24 kB 00:00 2024-03-29T10:38:15.461 INFO:teuthology.orchestra.run.smithi057.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 112 kB/s | 18 kB 00:00 2024-03-29T10:38:15.528 INFO:teuthology.orchestra.run.smithi057.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.0 MB/s | 132 kB 00:00 2024-03-29T10:38:15.553 INFO:teuthology.orchestra.run.smithi057.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 174 kB 00:00 2024-03-29T10:38:15.578 INFO:teuthology.orchestra.run.smithi057.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T10:38:15.604 INFO:teuthology.orchestra.run.smithi057.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 858 kB/s | 43 kB 00:00 2024-03-29T10:38:15.629 INFO:teuthology.orchestra.run.smithi057.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 159 kB/s | 52 kB 00:00 2024-03-29T10:38:15.771 INFO:teuthology.orchestra.run.smithi057.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 90 kB/s | 15 kB 00:00 2024-03-29T10:38:15.796 INFO:teuthology.orchestra.run.smithi057.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 148 kB/s | 32 kB 00:00 2024-03-29T10:38:15.921 INFO:teuthology.orchestra.run.smithi057.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 473 kB/s | 59 kB 00:00 2024-03-29T10:38:15.947 INFO:teuthology.orchestra.run.smithi057.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 638 kB/s | 203 kB 00:00 2024-03-29T10:38:16.038 INFO:teuthology.orchestra.run.smithi057.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.8 MB/s | 1.0 MB 00:00 2024-03-29T10:38:16.064 INFO:teuthology.orchestra.run.smithi057.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 631 kB/s | 90 kB 00:00 2024-03-29T10:38:16.065 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:16.066 INFO:teuthology.orchestra.run.smithi057.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2024-03-29T10:38:16.066 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-03-29T10:38:16.080 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-03-29T10:38:16.080 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-03-29T10:38:16.175 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-03-29T10:38:16.176 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-03-29T10:38:16.707 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-03-29T10:38:16.885 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:38:16.950 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:38:17.005 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:38:17.042 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:38:17.084 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:38:17.190 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:38:17.306 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:38:17.360 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:38:18.118 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:38:18.164 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:38:18.198 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:38:18.255 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:38:18.292 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:38:19.040 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:38:19.040 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:38:19.040 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:38:19.040 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:38:19.040 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:38:19.040 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:38:19.041 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:38:19.041 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:38:19.041 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:38:19.041 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:38:19.041 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:38:19.041 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:38:19.041 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:38:19.404 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:38:19.404 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:19.404 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-03-29T10:38:19.404 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:38:19.404 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:38:19.404 INFO:teuthology.orchestra.run.smithi057.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:38:19.404 INFO:teuthology.orchestra.run.smithi057.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:38:19.404 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:38:19.404 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:38:19.404 INFO:teuthology.orchestra.run.smithi057.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:38:19.404 INFO:teuthology.orchestra.run.smithi057.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:38:19.405 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:38:19.405 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:38:19.405 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:38:19.405 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:38:19.405 INFO:teuthology.orchestra.run.smithi057.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:38:19.405 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:19.405 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:38:19.636 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:38:20.216 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:31 ago on Fri 29 Mar 2024 10:36:49 AM UTC. 2024-03-29T10:38:20.376 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:38:20.377 INFO:teuthology.orchestra.run.smithi057.stdout:======================================================================================== 2024-03-29T10:38:20.377 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2024-03-29T10:38:20.377 INFO:teuthology.orchestra.run.smithi057.stdout:======================================================================================== 2024-03-29T10:38:20.377 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-03-29T10:38:20.377 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:38:20.377 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2024-03-29T10:38:20.378 INFO:teuthology.orchestra.run.smithi057.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:38:20.378 INFO:teuthology.orchestra.run.smithi057.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:38:20.378 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:38:20.378 INFO:teuthology.orchestra.run.smithi057.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:38:20.378 INFO:teuthology.orchestra.run.smithi057.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:38:20.378 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:20.378 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-03-29T10:38:20.378 INFO:teuthology.orchestra.run.smithi057.stdout:======================================================================================== 2024-03-29T10:38:20.378 INFO:teuthology.orchestra.run.smithi057.stdout:Install 6 Packages 2024-03-29T10:38:20.378 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:20.379 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 6.8 M 2024-03-29T10:38:20.379 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 33 M 2024-03-29T10:38:20.379 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-03-29T10:38:20.807 INFO:teuthology.orchestra.run.smithi057.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-03-29T10:38:20.907 INFO:teuthology.orchestra.run.smithi057.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-29T10:38:20.932 INFO:teuthology.orchestra.run.smithi057.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 463 kB/s | 138 kB 00:00 2024-03-29T10:38:21.258 INFO:teuthology.orchestra.run.smithi057.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 165 kB/s | 58 kB 00:00 2024-03-29T10:38:21.283 INFO:teuthology.orchestra.run.smithi057.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 146 kB/s | 51 kB 00:00 2024-03-29T10:38:21.450 INFO:teuthology.orchestra.run.smithi057.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 856 kB/s | 548 kB 00:00 2024-03-29T10:38:21.451 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:21.452 INFO:teuthology.orchestra.run.smithi057.stdout:Total 6.3 MB/s | 6.8 MB 00:01 2024-03-29T10:38:21.466 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-03-29T10:38:21.478 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-03-29T10:38:21.478 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-03-29T10:38:21.571 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-03-29T10:38:21.572 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-03-29T10:38:21.867 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-03-29T10:38:22.048 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:38:22.102 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:38:22.663 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:38:22.739 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:38:22.809 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:38:22.838 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:38:23.602 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:38:23.602 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:38:23.602 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:38:23.602 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:38:23.602 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:38:23.602 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:38:23.911 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:38:23.911 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:23.911 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-03-29T10:38:23.911 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:38:23.911 INFO:teuthology.orchestra.run.smithi057.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:38:23.911 INFO:teuthology.orchestra.run.smithi057.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:38:23.911 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:38:23.912 INFO:teuthology.orchestra.run.smithi057.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:38:23.912 INFO:teuthology.orchestra.run.smithi057.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:38:23.912 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:23.912 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:38:24.139 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-fuse 2024-03-29T10:38:24.712 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 10:36:49 AM UTC. 2024-03-29T10:38:24.869 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:38:24.869 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:38:24.869 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2024-03-29T10:38:24.869 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:38:24.869 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-03-29T10:38:24.869 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:38:24.869 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2024-03-29T10:38:24.870 INFO:teuthology.orchestra.run.smithi057.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:38:24.870 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:24.870 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-03-29T10:38:24.870 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:38:24.870 INFO:teuthology.orchestra.run.smithi057.stdout:Install 2 Packages 2024-03-29T10:38:24.870 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:24.870 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 964 k 2024-03-29T10:38:24.871 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 2.8 M 2024-03-29T10:38:24.871 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-03-29T10:38:25.017 INFO:teuthology.orchestra.run.smithi057.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 537 kB/s | 79 kB 00:00 2024-03-29T10:38:25.176 INFO:teuthology.orchestra.run.smithi057.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.8 MB/s | 885 kB 00:00 2024-03-29T10:38:25.177 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:25.177 INFO:teuthology.orchestra.run.smithi057.stdout:Total 3.1 MB/s | 964 kB 00:00 2024-03-29T10:38:25.177 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-03-29T10:38:25.199 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-03-29T10:38:25.199 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-03-29T10:38:25.275 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-03-29T10:38:25.275 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-03-29T10:38:25.457 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-03-29T10:38:25.597 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:38:25.669 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:38:26.377 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:38:26.377 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:38:26.675 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:38:26.675 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:26.675 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-03-29T10:38:26.675 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:38:26.675 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:26.676 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:38:26.939 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-volume 2024-03-29T10:38:27.522 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 10:36:49 AM UTC. 2024-03-29T10:38:27.681 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:38:27.682 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:38:27.682 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2024-03-29T10:38:27.682 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:38:27.682 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-03-29T10:38:27.682 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:38:27.682 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2024-03-29T10:38:27.682 INFO:teuthology.orchestra.run.smithi057.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:38:27.682 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:27.682 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-03-29T10:38:27.683 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:38:27.683 INFO:teuthology.orchestra.run.smithi057.stdout:Install 2 Packages 2024-03-29T10:38:27.683 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:27.683 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 332 k 2024-03-29T10:38:27.683 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 1.5 M 2024-03-29T10:38:27.683 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-03-29T10:38:27.815 INFO:teuthology.orchestra.run.smithi057.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 525 kB/s | 69 kB 00:00 2024-03-29T10:38:28.098 INFO:teuthology.orchestra.run.smithi057.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 636 kB/s | 264 kB 00:00 2024-03-29T10:38:28.099 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:28.100 INFO:teuthology.orchestra.run.smithi057.stdout:Total 797 kB/s | 332 kB 00:00 2024-03-29T10:38:28.100 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-03-29T10:38:28.109 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-03-29T10:38:28.109 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-03-29T10:38:28.151 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-03-29T10:38:28.152 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-03-29T10:38:28.301 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-03-29T10:38:28.472 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:38:28.510 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:38:28.525 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:38:28.525 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:38:28.525 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:38:28.525 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:29.206 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:38:29.594 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:38:29.594 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:29.594 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-03-29T10:38:29.594 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:38:29.594 INFO:teuthology.orchestra.run.smithi057.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:38:29.594 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:29.594 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:38:29.940 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install librados-devel 2024-03-29T10:38:30.525 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:36:49 AM UTC. 2024-03-29T10:38:30.684 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:38:30.685 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:38:30.685 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repo Size 2024-03-29T10:38:30.685 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:38:30.686 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-03-29T10:38:30.686 INFO:teuthology.orchestra.run.smithi057.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:38:30.686 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:30.686 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-03-29T10:38:30.686 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:38:30.686 INFO:teuthology.orchestra.run.smithi057.stdout:Install 1 Package 2024-03-29T10:38:30.686 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:30.686 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 127 k 2024-03-29T10:38:30.686 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 456 k 2024-03-29T10:38:30.686 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-03-29T10:38:30.995 INFO:teuthology.orchestra.run.smithi057.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 410 kB/s | 127 kB 00:00 2024-03-29T10:38:30.995 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:30.995 INFO:teuthology.orchestra.run.smithi057.stdout:Total 408 kB/s | 127 kB 00:00 2024-03-29T10:38:30.996 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-03-29T10:38:31.001 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-03-29T10:38:31.001 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-03-29T10:38:31.060 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-03-29T10:38:31.060 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-03-29T10:38:31.215 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-03-29T10:38:31.320 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:31.758 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:32.082 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:32.082 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:32.082 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-03-29T10:38:32.082 INFO:teuthology.orchestra.run.smithi057.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:32.082 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:32.083 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:38:32.322 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install libcephfs2 2024-03-29T10:38:32.904 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:36:49 AM UTC. 2024-03-29T10:38:33.009 INFO:teuthology.orchestra.run.smithi057.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:38:33.065 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:38:33.067 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2024-03-29T10:38:33.067 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:38:33.152 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install libcephfs-devel 2024-03-29T10:38:33.729 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:36:49 AM UTC. 2024-03-29T10:38:33.885 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:38:33.886 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:38:33.886 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repo Size 2024-03-29T10:38:33.886 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:38:33.887 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-03-29T10:38:33.887 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:38:33.887 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:33.887 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-03-29T10:38:33.887 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:38:33.887 INFO:teuthology.orchestra.run.smithi057.stdout:Install 1 Package 2024-03-29T10:38:33.887 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:33.887 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 31 k 2024-03-29T10:38:33.887 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 140 k 2024-03-29T10:38:33.887 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-03-29T10:38:34.172 INFO:teuthology.orchestra.run.smithi057.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 109 kB/s | 31 kB 00:00 2024-03-29T10:38:34.173 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:34.173 INFO:teuthology.orchestra.run.smithi057.stdout:Total 109 kB/s | 31 kB 00:00 2024-03-29T10:38:34.173 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-03-29T10:38:34.176 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-03-29T10:38:34.177 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-03-29T10:38:34.206 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-03-29T10:38:34.206 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-03-29T10:38:34.354 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-03-29T10:38:34.474 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:34.863 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:35.139 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:35.139 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:35.139 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-03-29T10:38:35.139 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:35.140 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:35.140 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:38:35.287 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install librados2 2024-03-29T10:38:35.858 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 10:36:49 AM UTC. 2024-03-29T10:38:35.958 INFO:teuthology.orchestra.run.smithi057.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:38:36.013 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:38:36.014 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2024-03-29T10:38:36.014 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:38:36.067 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install librbd1 2024-03-29T10:38:36.640 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 10:36:49 AM UTC. 2024-03-29T10:38:36.740 INFO:teuthology.orchestra.run.smithi057.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:38:36.795 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:38:36.796 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2024-03-29T10:38:36.796 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:38:36.848 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install python3-rados 2024-03-29T10:38:37.428 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 10:36:49 AM UTC. 2024-03-29T10:38:37.529 INFO:teuthology.orchestra.run.smithi057.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:38:37.584 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:38:37.586 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2024-03-29T10:38:37.586 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:38:37.634 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install python3-rgw 2024-03-29T10:38:38.212 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 10:36:49 AM UTC. 2024-03-29T10:38:38.313 INFO:teuthology.orchestra.run.smithi057.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:38:38.368 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:38:38.369 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2024-03-29T10:38:38.370 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:38:38.417 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install python3-cephfs 2024-03-29T10:38:38.995 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 10:36:49 AM UTC. 2024-03-29T10:38:39.096 INFO:teuthology.orchestra.run.smithi057.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:38:39.151 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:38:39.152 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2024-03-29T10:38:39.153 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:38:39.207 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install python3-rbd 2024-03-29T10:38:39.784 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 10:36:49 AM UTC. 2024-03-29T10:38:39.886 INFO:teuthology.orchestra.run.smithi057.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:38:39.941 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:38:39.942 INFO:teuthology.orchestra.run.smithi057.stdout:Nothing to do. 2024-03-29T10:38:39.942 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:38:39.994 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install rbd-fuse 2024-03-29T10:38:40.569 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Mar 2024 10:36:49 AM UTC. 2024-03-29T10:38:40.724 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:38:40.725 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:38:40.726 INFO:teuthology.orchestra.run.smithi057.stdout: Package Architecture Version Repository Size 2024-03-29T10:38:40.726 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:38:40.726 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-03-29T10:38:40.726 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:38:40.726 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:40.726 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-03-29T10:38:40.726 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:38:40.727 INFO:teuthology.orchestra.run.smithi057.stdout:Install 1 Package 2024-03-29T10:38:40.727 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:40.727 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 85 k 2024-03-29T10:38:40.727 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 231 k 2024-03-29T10:38:40.727 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-03-29T10:38:40.917 INFO:teuthology.orchestra.run.smithi057.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 447 kB/s | 85 kB 00:00 2024-03-29T10:38:40.917 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:40.918 INFO:teuthology.orchestra.run.smithi057.stdout:Total 444 kB/s | 85 kB 00:00 2024-03-29T10:38:40.918 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-03-29T10:38:40.925 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-03-29T10:38:40.925 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-03-29T10:38:40.982 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-03-29T10:38:40.982 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-03-29T10:38:41.127 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-03-29T10:38:41.243 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:41.593 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:41.898 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:41.898 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:41.898 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-03-29T10:38:41.898 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:41.898 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:41.898 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:38:42.160 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install rbd-mirror 2024-03-29T10:38:42.755 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 10:36:49 AM UTC. 2024-03-29T10:38:42.916 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:38:42.917 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:38:42.917 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repo Size 2024-03-29T10:38:42.917 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:38:42.917 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-03-29T10:38:42.917 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:38:42.917 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:42.917 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-03-29T10:38:42.917 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:38:42.917 INFO:teuthology.orchestra.run.smithi057.stdout:Install 1 Package 2024-03-29T10:38:42.918 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:42.918 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 3.1 M 2024-03-29T10:38:42.918 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 13 M 2024-03-29T10:38:42.918 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-03-29T10:38:45.752 INFO:teuthology.orchestra.run.smithi057.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 1.1 MB/s | 3.1 MB 00:02 2024-03-29T10:38:45.753 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:45.754 INFO:teuthology.orchestra.run.smithi057.stdout:Total 1.1 MB/s | 3.1 MB 00:02 2024-03-29T10:38:45.754 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-03-29T10:38:45.766 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-03-29T10:38:45.766 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-03-29T10:38:45.846 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-03-29T10:38:45.846 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-03-29T10:38:46.309 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-03-29T10:38:46.370 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:46.395 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:46.396 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:38:46.396 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:38:46.396 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:38:46.396 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:38:46.397 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:47.496 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:47.496 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:47.496 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-03-29T10:38:47.496 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:47.496 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:47.496 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:38:47.722 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install rbd-nbd 2024-03-29T10:38:48.299 INFO:teuthology.orchestra.run.smithi057.stdout:Last metadata expiration check: 0:01:59 ago on Fri 29 Mar 2024 10:36:49 AM UTC. 2024-03-29T10:38:48.454 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-29T10:38:48.455 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:38:48.455 INFO:teuthology.orchestra.run.smithi057.stdout: Package Architecture Version Repository Size 2024-03-29T10:38:48.455 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:38:48.455 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-03-29T10:38:48.455 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:38:48.456 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:48.456 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-03-29T10:38:48.456 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-29T10:38:48.456 INFO:teuthology.orchestra.run.smithi057.stdout:Install 1 Package 2024-03-29T10:38:48.456 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:48.456 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 171 k 2024-03-29T10:38:48.456 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 494 k 2024-03-29T10:38:48.456 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-03-29T10:38:48.889 INFO:teuthology.orchestra.run.smithi057.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 397 kB/s | 171 kB 00:00 2024-03-29T10:38:48.889 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:38:48.889 INFO:teuthology.orchestra.run.smithi057.stdout:Total 396 kB/s | 171 kB 00:00 2024-03-29T10:38:48.889 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-03-29T10:38:48.897 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-03-29T10:38:48.897 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-03-29T10:38:48.956 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-03-29T10:38:48.956 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-03-29T10:38:49.102 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-03-29T10:38:49.206 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:49.659 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:49.988 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:38:49.988 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:49.988 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-03-29T10:38:49.988 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:38:49.988 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:49.988 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-29T10:38:50.243 DEBUG:teuthology.parallel:result is None 2024-03-29T10:38:50.244 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:38:50.465 DEBUG:teuthology.orchestra.run.smithi057:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:38:50.487 INFO:teuthology.orchestra.run.smithi057.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:38:50.487 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:38:50.487 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:38:50.489 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:38:50.702 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:38:50.727 INFO:teuthology.orchestra.run.smithi179.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:38:50.727 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:38:50.727 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:38:50.729 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T10:38:50.729 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-29T10:38:50.729 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:38:50.758 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T10:38:50.758 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:38:50.799 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T10:38:50.799 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-29T10:38:50.799 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:38:50.827 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:38:50.895 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T10:38:50.895 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:38:50.935 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:38:51.003 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T10:38:51.003 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-29T10:38:51.003 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:38:51.031 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:38:51.097 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T10:38:51.098 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:38:51.125 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:38:51.192 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T10:38:51.193 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-29T10:38:51.193 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:38:51.220 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:38:51.287 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T10:38:51.288 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:38:51.314 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:38:51.383 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T10:38:51.484 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T10:38:51.485 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 777 /var/log/ceph 2024-03-29T10:38:51.487 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 777 /var/log/ceph 2024-03-29T10:38:51.515 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T10:38:51.515 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:38:51.555 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:38:51.583 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T10:38:51.583 DEBUG:teuthology.orchestra.run.smithi057:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:38:51.622 DEBUG:teuthology.orchestra.run.smithi179:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:38:51.660 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T10:38:51.660 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T10:38:51.661 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622641', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622641', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi057.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOIvQcBjWXDczcrEinUPtbVNMsWv8poI/HVsMsTY3lZhdJ/G2+kuaCcn8AlPIGoyBeouP2tDxUXV2DDKLqHqJMU=', 'smithi179.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOAEcOUl39u6xHmX9AOi+6H5hugRXuFb1wwZYRXKWzwhnLEYf6371TH0sVoDdYEUpoblQup/LyAf0MeJ87SVye4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T10:38:51.661 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:38:51.690 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:38:51.705 DEBUG:teuthology.orchestra.run.smithi057:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:38:51.745 DEBUG:teuthology.orchestra.run.smithi179:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:38:51.774 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-29T10:38:51.775 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:38:51.825 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:38:51.825 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_1 2024-03-29T10:38:51.879 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:38:51.879 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:38:51.879 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-29T10:38:51.879 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:38:51.880 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:38:51.880 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2024-03-29 10:38:46.655852809 +0000 2024-03-29T10:38:51.880 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2024-03-29 10:33:21.614955668 +0000 2024-03-29T10:38:51.880 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2024-03-29 10:33:21.614955668 +0000 2024-03-29T10:38:51.880 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2024-03-29 10:33:21.614955668 +0000 2024-03-29T10:38:51.880 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:38:51.945 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2024-03-29T10:38:51.945 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2024-03-29T10:38:51.945 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000284821 s, 1.8 MB/s 2024-03-29T10:38:51.946 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:38:52.002 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_2 2024-03-29T10:38:52.057 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:38:52.057 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:38:52.057 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T10:38:52.057 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:38:52.057 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:38:52.057 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2024-03-29 10:38:46.655852809 +0000 2024-03-29T10:38:52.057 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2024-03-29 10:33:21.957950675 +0000 2024-03-29T10:38:52.057 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2024-03-29 10:33:21.957950675 +0000 2024-03-29T10:38:52.057 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2024-03-29 10:33:21.957950675 +0000 2024-03-29T10:38:52.058 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:38:52.123 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2024-03-29T10:38:52.123 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2024-03-29T10:38:52.123 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000298858 s, 1.7 MB/s 2024-03-29T10:38:52.124 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:38:52.179 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_3 2024-03-29T10:38:52.234 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:38:52.234 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:38:52.234 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-29T10:38:52.234 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:38:52.234 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:38:52.234 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2024-03-29 10:38:46.655852809 +0000 2024-03-29T10:38:52.234 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2024-03-29 10:33:22.305945609 +0000 2024-03-29T10:38:52.234 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2024-03-29 10:33:22.305945609 +0000 2024-03-29T10:38:52.234 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2024-03-29 10:33:22.305945609 +0000 2024-03-29T10:38:52.235 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:38:52.299 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2024-03-29T10:38:52.300 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2024-03-29T10:38:52.300 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000291431 s, 1.8 MB/s 2024-03-29T10:38:52.301 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:38:52.357 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_4 2024-03-29T10:38:52.412 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:38:52.412 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:38:52.412 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-29T10:38:52.412 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:38:52.412 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:38:52.412 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2024-03-29 10:38:46.656852794 +0000 2024-03-29T10:38:52.412 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2024-03-29 10:33:22.668940325 +0000 2024-03-29T10:38:52.412 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2024-03-29 10:33:22.669940311 +0000 2024-03-29T10:38:52.412 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2024-03-29 10:33:22.668940325 +0000 2024-03-29T10:38:52.413 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:38:52.477 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2024-03-29T10:38:52.478 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2024-03-29T10:38:52.478 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000253735 s, 2.0 MB/s 2024-03-29T10:38:52.479 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:38:52.533 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:52.533 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T10:38:52.534 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:38:52.550 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:38:52.550 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_1 2024-03-29T10:38:52.606 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:38:52.606 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:38:52.606 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T10:38:52.606 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:38:52.606 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:38:52.607 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-03-29 10:35:59.706665306 +0000 2024-03-29T10:38:52.607 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-03-29 10:33:26.612486298 +0000 2024-03-29T10:38:52.607 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-03-29 10:33:26.612486298 +0000 2024-03-29T10:38:52.607 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-03-29 10:33:26.612486298 +0000 2024-03-29T10:38:52.607 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:38:52.672 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-03-29T10:38:52.673 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-03-29T10:38:52.673 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000251966 s, 2.0 MB/s 2024-03-29T10:38:52.675 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:38:52.729 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_2 2024-03-29T10:38:52.784 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:38:52.784 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:38:52.784 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T10:38:52.784 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:38:52.785 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:38:52.785 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-03-29 10:35:59.706665306 +0000 2024-03-29T10:38:52.785 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-03-29 10:33:26.955478725 +0000 2024-03-29T10:38:52.785 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-03-29 10:33:26.955478725 +0000 2024-03-29T10:38:52.785 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-03-29 10:33:26.955478725 +0000 2024-03-29T10:38:52.785 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:38:52.850 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-03-29T10:38:52.850 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-03-29T10:38:52.850 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000191942 s, 2.7 MB/s 2024-03-29T10:38:52.851 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:38:52.906 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_3 2024-03-29T10:38:52.962 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:38:52.962 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:38:52.962 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T10:38:52.962 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:38:52.962 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:38:52.962 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-03-29 10:35:59.707665284 +0000 2024-03-29T10:38:52.962 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-03-29 10:33:27.288471373 +0000 2024-03-29T10:38:52.963 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-03-29 10:33:27.288471373 +0000 2024-03-29T10:38:52.963 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-03-29 10:33:27.288471373 +0000 2024-03-29T10:38:52.963 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:38:53.027 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-03-29T10:38:53.027 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-03-29T10:38:53.027 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000190831 s, 2.7 MB/s 2024-03-29T10:38:53.028 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:38:53.083 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_4 2024-03-29T10:38:53.138 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:38:53.138 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:38:53.138 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T10:38:53.138 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:38:53.139 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:38:53.139 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-03-29 10:35:59.707665284 +0000 2024-03-29T10:38:53.139 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-03-29 10:33:27.601464462 +0000 2024-03-29T10:38:53.139 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-03-29 10:33:27.601464462 +0000 2024-03-29T10:38:53.139 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-03-29 10:33:27.601464462 +0000 2024-03-29T10:38:53.139 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:38:53.204 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-03-29T10:38:53.204 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-03-29T10:38:53.204 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000228984 s, 2.2 MB/s 2024-03-29T10:38:53.205 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:38:53.262 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:53.262 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi057.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi179.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:38:53.262 INFO:tasks.ceph:Generating config... 2024-03-29T10:38:53.263 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T10:38:53.263 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T10:38:53.263 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T10:38:53.263 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T10:38:53.263 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T10:38:53.263 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T10:38:53.264 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T10:38:53.264 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T10:38:53.264 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T10:38:53.264 INFO:tasks.ceph:[global] ms type = async 2024-03-29T10:38:53.264 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T10:38:53.264 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T10:38:53.264 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T10:38:53.264 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T10:38:53.264 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T10:38:53.264 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T10:38:53.264 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T10:38:53.264 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T10:38:53.264 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T10:38:53.264 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T10:38:53.265 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T10:38:53.265 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T10:38:53.265 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T10:38:53.265 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T10:38:53.265 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T10:38:53.265 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T10:38:53.265 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T10:38:53.265 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T10:38:53.265 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T10:38:53.265 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T10:38:53.265 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T10:38:53.265 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T10:38:53.265 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T10:38:53.265 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T10:38:53.265 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T10:38:53.266 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T10:38:53.266 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T10:38:53.266 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T10:38:53.266 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T10:38:53.266 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T10:38:53.266 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T10:38:53.266 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T10:38:53.266 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T10:38:53.266 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T10:38:53.266 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T10:38:53.266 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T10:38:53.266 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T10:38:53.266 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T10:38:53.267 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T10:38:53.267 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T10:38:53.316 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T10:38:53.319 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T10:38:53.365 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:38:53.432 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.57:6789]'), ('mon.c', '[v1:172.21.15.57:6790]'), ('mon.b', '[v1:172.21.15.179:6789]')] 2024-03-29T10:38:53.432 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.57:6789],[v1:172.21.15.57:6790],[v1:172.21.15.179:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T10:38:53.433 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-29T10:38:53.433 DEBUG:teuthology.orchestra.run.smithi057:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:38:53.488 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.57:6789]' --addv c '[v1:172.21.15.57:6790]' --addv b '[v1:172.21.15.179:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:53.574 INFO:teuthology.orchestra.run.smithi057.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:53.574 INFO:teuthology.orchestra.run.smithi057.stdout:monmaptool: generated fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:38:53.574 INFO:teuthology.orchestra.run.smithi057.stdout:setting min_mon_release = quincy 2024-03-29T10:38:53.574 INFO:teuthology.orchestra.run.smithi057.stdout:epoch 0 2024-03-29T10:38:53.574 INFO:teuthology.orchestra.run.smithi057.stdout:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:38:53.574 INFO:teuthology.orchestra.run.smithi057.stdout:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:38:53.574 INFO:teuthology.orchestra.run.smithi057.stdout:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:38:53.574 INFO:teuthology.orchestra.run.smithi057.stdout:min_mon_release 17 (quincy) 2024-03-29T10:38:53.574 INFO:teuthology.orchestra.run.smithi057.stdout:election_strategy: 1 2024-03-29T10:38:53.574 INFO:teuthology.orchestra.run.smithi057.stdout:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:38:53.574 INFO:teuthology.orchestra.run.smithi057.stdout:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:38:53.575 INFO:teuthology.orchestra.run.smithi057.stdout:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:38:53.575 INFO:teuthology.orchestra.run.smithi057.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T10:38:53.576 DEBUG:teuthology.orchestra.run.smithi057:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:38:53.590 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID c698cedc-432d-480a-94d2-408defa62225... 2024-03-29T10:38:53.591 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:38:53.633 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:38:53.680 INFO:teuthology.orchestra.run.smithi057.stdout:[global] 2024-03-29T10:38:53.681 INFO:teuthology.orchestra.run.smithi057.stdout: chdir = "" 2024-03-29T10:38:53.681 INFO:teuthology.orchestra.run.smithi057.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:38:53.681 INFO:teuthology.orchestra.run.smithi057.stdout: auth supported = cephx 2024-03-29T10:38:53.681 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.681 INFO:teuthology.orchestra.run.smithi057.stdout: filestore xattr use omap = true 2024-03-29T10:38:53.681 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.681 INFO:teuthology.orchestra.run.smithi057.stdout: mon clock drift allowed = 1.000 2024-03-29T10:38:53.681 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.681 INFO:teuthology.orchestra.run.smithi057.stdout: osd crush chooseleaf type = 0 2024-03-29T10:38:53.681 INFO:teuthology.orchestra.run.smithi057.stdout: auth debug = true 2024-03-29T10:38:53.681 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.681 INFO:teuthology.orchestra.run.smithi057.stdout: ms die on old message = true 2024-03-29T10:38:53.682 INFO:teuthology.orchestra.run.smithi057.stdout: ms die on bug = true 2024-03-29T10:38:53.682 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.682 INFO:teuthology.orchestra.run.smithi057.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:38:53.682 INFO:teuthology.orchestra.run.smithi057.stdout: mon pg warn max object skew = 0 2024-03-29T10:38:53.682 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.682 INFO:teuthology.orchestra.run.smithi057.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:38:53.682 INFO:teuthology.orchestra.run.smithi057.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:38:53.682 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.682 INFO:teuthology.orchestra.run.smithi057.stdout: osd pool default size = 2 2024-03-29T10:38:53.682 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.682 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd allow primary affinity = true 2024-03-29T10:38:53.682 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd allow pg remap = true 2024-03-29T10:38:53.682 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:38:53.683 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:38:53.683 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on no sortbitwise = false 2024-03-29T10:38:53.683 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:38:53.683 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on too few osds = false 2024-03-29T10:38:53.683 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:38:53.683 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:38:53.683 INFO:teuthology.orchestra.run.smithi057.stdout: mon_allow_pool_size_one = true 2024-03-29T10:38:53.683 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.683 INFO:teuthology.orchestra.run.smithi057.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:38:53.683 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.683 INFO:teuthology.orchestra.run.smithi057.stdout: osd default data pool replay window = 5 2024-03-29T10:38:53.683 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.683 INFO:teuthology.orchestra.run.smithi057.stdout: mon allow pool delete = true 2024-03-29T10:38:53.683 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.683 INFO:teuthology.orchestra.run.smithi057.stdout: mon cluster log file level = debug 2024-03-29T10:38:53.684 INFO:teuthology.orchestra.run.smithi057.stdout: debug asserts on shutdown = true 2024-03-29T10:38:53.684 INFO:teuthology.orchestra.run.smithi057.stdout: mon health detail to clog = false 2024-03-29T10:38:53.684 INFO:teuthology.orchestra.run.smithi057.stdout: mon host = "[v1:172.21.15.57:6789],[v1:172.21.15.57:6790],[v1:172.21.15.179:6789]" 2024-03-29T10:38:53.684 INFO:teuthology.orchestra.run.smithi057.stdout: lockdep = True 2024-03-29T10:38:53.684 INFO:teuthology.orchestra.run.smithi057.stdout: mon client directed command retry = 5 2024-03-29T10:38:53.684 INFO:teuthology.orchestra.run.smithi057.stdout: mon election default strategy = 1 2024-03-29T10:38:53.684 INFO:teuthology.orchestra.run.smithi057.stdout: ms bind msgr2 = False 2024-03-29T10:38:53.684 INFO:teuthology.orchestra.run.smithi057.stdout: ms inject socket failures = 5000 2024-03-29T10:38:53.684 INFO:teuthology.orchestra.run.smithi057.stdout: ms type = async 2024-03-29T10:38:53.684 INFO:teuthology.orchestra.run.smithi057.stdout: fsid = c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:38:53.684 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.684 INFO:teuthology.orchestra.run.smithi057.stdout:[osd] 2024-03-29T10:38:53.684 INFO:teuthology.orchestra.run.smithi057.stdout: osd journal size = 100 2024-03-29T10:38:53.684 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.684 INFO:teuthology.orchestra.run.smithi057.stdout: osd scrub load threshold = 5.0 2024-03-29T10:38:53.685 INFO:teuthology.orchestra.run.smithi057.stdout: osd scrub max interval = 600 2024-03-29T10:38:53.685 INFO:teuthology.orchestra.run.smithi057.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:38:53.685 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.685 INFO:teuthology.orchestra.run.smithi057.stdout: osd recover clone overlap = true 2024-03-29T10:38:53.685 INFO:teuthology.orchestra.run.smithi057.stdout: osd recovery max chunk = 1048576 2024-03-29T10:38:53.685 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.685 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug shutdown = true 2024-03-29T10:38:53.685 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug op order = true 2024-03-29T10:38:53.685 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug verify stray on activate = true 2024-03-29T10:38:53.685 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug trim objects = true 2024-03-29T10:38:53.685 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.685 INFO:teuthology.orchestra.run.smithi057.stdout: osd open classes on start = true 2024-03-29T10:38:53.685 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug pg log writeout = true 2024-03-29T10:38:53.685 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.685 INFO:teuthology.orchestra.run.smithi057.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:38:53.686 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.686 INFO:teuthology.orchestra.run.smithi057.stdout: osd map max advance = 10 2024-03-29T10:38:53.686 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.686 INFO:teuthology.orchestra.run.smithi057.stdout: journal zero on create = true 2024-03-29T10:38:53.686 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.686 INFO:teuthology.orchestra.run.smithi057.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:38:53.686 INFO:teuthology.orchestra.run.smithi057.stdout: filestore apply finisher threads = 3 2024-03-29T10:38:53.686 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.686 INFO:teuthology.orchestra.run.smithi057.stdout: bdev debug aio = true 2024-03-29T10:38:53.686 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug misdirected ops = true 2024-03-29T10:38:53.686 INFO:teuthology.orchestra.run.smithi057.stdout: bluestore block size = 96636764160 2024-03-29T10:38:53.686 INFO:teuthology.orchestra.run.smithi057.stdout: bluestore compression algorithm = zstd 2024-03-29T10:38:53.686 INFO:teuthology.orchestra.run.smithi057.stdout: bluestore compression mode = aggressive 2024-03-29T10:38:53.686 INFO:teuthology.orchestra.run.smithi057.stdout: bluestore fsck on mount = True 2024-03-29T10:38:53.687 INFO:teuthology.orchestra.run.smithi057.stdout: bluestore zero block detection = True 2024-03-29T10:38:53.687 INFO:teuthology.orchestra.run.smithi057.stdout: debug bluefs = 20 2024-03-29T10:38:53.687 INFO:teuthology.orchestra.run.smithi057.stdout: debug bluestore = 20 2024-03-29T10:38:53.687 INFO:teuthology.orchestra.run.smithi057.stdout: debug monc = 20 2024-03-29T10:38:53.687 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2024-03-29T10:38:53.687 INFO:teuthology.orchestra.run.smithi057.stdout: debug osd = 20 2024-03-29T10:38:53.687 INFO:teuthology.orchestra.run.smithi057.stdout: debug rocksdb = 10 2024-03-29T10:38:53.687 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:38:53.687 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd full ratio = 0.9 2024-03-29T10:38:53.687 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:38:53.687 INFO:teuthology.orchestra.run.smithi057.stdout: osd class default list = * 2024-03-29T10:38:53.687 INFO:teuthology.orchestra.run.smithi057.stdout: osd class load list = * 2024-03-29T10:38:53.687 INFO:teuthology.orchestra.run.smithi057.stdout: osd client watch timeout = 120 2024-03-29T10:38:53.687 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug verify cached snaps = True 2024-03-29T10:38:53.687 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug verify missing on start = True 2024-03-29T10:38:53.688 INFO:teuthology.orchestra.run.smithi057.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:38:53.688 INFO:teuthology.orchestra.run.smithi057.stdout: osd mclock override recovery settings = True 2024-03-29T10:38:53.688 INFO:teuthology.orchestra.run.smithi057.stdout: osd objectstore = bluestore 2024-03-29T10:38:53.688 INFO:teuthology.orchestra.run.smithi057.stdout: osd op queue = debug_random 2024-03-29T10:38:53.688 INFO:teuthology.orchestra.run.smithi057.stdout: osd op queue cut off = debug_random 2024-03-29T10:38:53.688 INFO:teuthology.orchestra.run.smithi057.stdout: osd shutdown pgref assert = True 2024-03-29T10:38:53.688 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.688 INFO:teuthology.orchestra.run.smithi057.stdout:[mgr] 2024-03-29T10:38:53.688 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2024-03-29T10:38:53.688 INFO:teuthology.orchestra.run.smithi057.stdout: debug mgr = 20 2024-03-29T10:38:53.688 INFO:teuthology.orchestra.run.smithi057.stdout: debug mon = 20 2024-03-29T10:38:53.688 INFO:teuthology.orchestra.run.smithi057.stdout: debug auth = 20 2024-03-29T10:38:53.688 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:38:53.688 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:38:53.688 INFO:teuthology.orchestra.run.smithi057.stdout: mgr/telemetry/nag = false 2024-03-29T10:38:53.689 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.689 INFO:teuthology.orchestra.run.smithi057.stdout:[mon] 2024-03-29T10:38:53.689 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2024-03-29T10:38:53.689 INFO:teuthology.orchestra.run.smithi057.stdout: debug mon = 20 2024-03-29T10:38:53.689 INFO:teuthology.orchestra.run.smithi057.stdout: debug paxos = 20 2024-03-29T10:38:53.689 INFO:teuthology.orchestra.run.smithi057.stdout: debug auth = 20 2024-03-29T10:38:53.689 INFO:teuthology.orchestra.run.smithi057.stdout: mon data avail warn = 5 2024-03-29T10:38:53.689 INFO:teuthology.orchestra.run.smithi057.stdout: mon mgr mkfs grace = 240 2024-03-29T10:38:53.689 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:38:53.689 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd reporter subtree level = osd 2024-03-29T10:38:53.689 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd prime pg temp = true 2024-03-29T10:38:53.689 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:38:53.689 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.689 INFO:teuthology.orchestra.run.smithi057.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:38:53.689 INFO:teuthology.orchestra.run.smithi057.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:38:53.690 INFO:teuthology.orchestra.run.smithi057.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:38:53.690 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.690 INFO:teuthology.orchestra.run.smithi057.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:38:53.690 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:38:53.690 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:38:53.690 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.690 INFO:teuthology.orchestra.run.smithi057.stdout: # 1m isn't quite enough 2024-03-29T10:38:53.690 INFO:teuthology.orchestra.run.smithi057.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:38:53.690 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.690 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:38:53.690 INFO:teuthology.orchestra.run.smithi057.stdout: mon min osdmap epochs = 50 2024-03-29T10:38:53.690 INFO:teuthology.orchestra.run.smithi057.stdout: mon osdmap full prune interval = 2 2024-03-29T10:38:53.690 INFO:teuthology.orchestra.run.smithi057.stdout: mon osdmap full prune min = 15 2024-03-29T10:38:53.690 INFO:teuthology.orchestra.run.smithi057.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:38:53.690 INFO:teuthology.orchestra.run.smithi057.stdout: mon scrub interval = 300 2024-03-29T10:38:53.691 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on pool no app = False 2024-03-29T10:38:53.691 INFO:teuthology.orchestra.run.smithi057.stdout: paxos service trim min = 10 2024-03-29T10:38:53.691 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:38:53.691 INFO:teuthology.orchestra.run.smithi057.stdout:[client] 2024-03-29T10:38:53.691 INFO:teuthology.orchestra.run.smithi057.stdout: rgw cache enabled = true 2024-03-29T10:38:53.691 INFO:teuthology.orchestra.run.smithi057.stdout: rgw enable ops log = true 2024-03-29T10:38:53.691 INFO:teuthology.orchestra.run.smithi057.stdout: rgw enable usage log = true 2024-03-29T10:38:53.691 INFO:teuthology.orchestra.run.smithi057.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:38:53.691 INFO:teuthology.orchestra.run.smithi057.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:38:53.691 INFO:teuthology.orchestra.run.smithi057.stdout: debug monc = 20 2024-03-29T10:38:53.691 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2024-03-29T10:38:53.691 INFO:teuthology.orchestra.run.smithi057.stdout: debug objecter = 20 2024-03-29T10:38:53.691 INFO:teuthology.orchestra.run.smithi057.stdout: debug rados = 20 2024-03-29T10:38:53.691 INFO:teuthology.orchestra.run.smithi057.stdout:[mon.a] 2024-03-29T10:38:53.692 INFO:teuthology.orchestra.run.smithi057.stdout:[mon.c] 2024-03-29T10:38:53.692 INFO:teuthology.orchestra.run.smithi057.stdout:[mon.b] 2024-03-29T10:38:53.692 INFO:teuthology.orchestra.run.smithi179.stdout:[global] 2024-03-29T10:38:53.692 INFO:teuthology.orchestra.run.smithi179.stdout: chdir = "" 2024-03-29T10:38:53.692 INFO:teuthology.orchestra.run.smithi179.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:38:53.692 INFO:teuthology.orchestra.run.smithi179.stdout: auth supported = cephx 2024-03-29T10:38:53.692 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.692 INFO:teuthology.orchestra.run.smithi179.stdout: filestore xattr use omap = true 2024-03-29T10:38:53.693 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.693 INFO:teuthology.orchestra.run.smithi179.stdout: mon clock drift allowed = 1.000 2024-03-29T10:38:53.693 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.693 INFO:teuthology.orchestra.run.smithi179.stdout: osd crush chooseleaf type = 0 2024-03-29T10:38:53.693 INFO:teuthology.orchestra.run.smithi179.stdout: auth debug = true 2024-03-29T10:38:53.693 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.693 INFO:teuthology.orchestra.run.smithi179.stdout: ms die on old message = true 2024-03-29T10:38:53.693 INFO:teuthology.orchestra.run.smithi179.stdout: ms die on bug = true 2024-03-29T10:38:53.693 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.693 INFO:teuthology.orchestra.run.smithi179.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:38:53.693 INFO:teuthology.orchestra.run.smithi179.stdout: mon pg warn max object skew = 0 2024-03-29T10:38:53.693 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.693 INFO:teuthology.orchestra.run.smithi179.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:38:53.693 INFO:teuthology.orchestra.run.smithi179.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:38:53.694 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.694 INFO:teuthology.orchestra.run.smithi179.stdout: osd pool default size = 2 2024-03-29T10:38:53.694 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.694 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd allow primary affinity = true 2024-03-29T10:38:53.694 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd allow pg remap = true 2024-03-29T10:38:53.694 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:38:53.694 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:38:53.694 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on no sortbitwise = false 2024-03-29T10:38:53.694 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:38:53.694 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on too few osds = false 2024-03-29T10:38:53.694 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:38:53.694 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:38:53.694 INFO:teuthology.orchestra.run.smithi179.stdout: mon_allow_pool_size_one = true 2024-03-29T10:38:53.694 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.695 INFO:teuthology.orchestra.run.smithi179.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:38:53.695 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.695 INFO:teuthology.orchestra.run.smithi179.stdout: osd default data pool replay window = 5 2024-03-29T10:38:53.695 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.695 INFO:teuthology.orchestra.run.smithi179.stdout: mon allow pool delete = true 2024-03-29T10:38:53.695 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.695 INFO:teuthology.orchestra.run.smithi179.stdout: mon cluster log file level = debug 2024-03-29T10:38:53.695 INFO:teuthology.orchestra.run.smithi179.stdout: debug asserts on shutdown = true 2024-03-29T10:38:53.695 INFO:teuthology.orchestra.run.smithi179.stdout: mon health detail to clog = false 2024-03-29T10:38:53.695 INFO:teuthology.orchestra.run.smithi179.stdout: mon host = "[v1:172.21.15.57:6789],[v1:172.21.15.57:6790],[v1:172.21.15.179:6789]" 2024-03-29T10:38:53.695 INFO:teuthology.orchestra.run.smithi179.stdout: lockdep = True 2024-03-29T10:38:53.695 INFO:teuthology.orchestra.run.smithi179.stdout: mon client directed command retry = 5 2024-03-29T10:38:53.695 INFO:teuthology.orchestra.run.smithi179.stdout: mon election default strategy = 1 2024-03-29T10:38:53.695 INFO:teuthology.orchestra.run.smithi179.stdout: ms bind msgr2 = False 2024-03-29T10:38:53.695 INFO:teuthology.orchestra.run.smithi179.stdout: ms inject socket failures = 5000 2024-03-29T10:38:53.696 INFO:teuthology.orchestra.run.smithi179.stdout: ms type = async 2024-03-29T10:38:53.696 INFO:teuthology.orchestra.run.smithi179.stdout: fsid = c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:38:53.696 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.696 INFO:teuthology.orchestra.run.smithi179.stdout:[osd] 2024-03-29T10:38:53.696 INFO:teuthology.orchestra.run.smithi179.stdout: osd journal size = 100 2024-03-29T10:38:53.696 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.696 INFO:teuthology.orchestra.run.smithi179.stdout: osd scrub load threshold = 5.0 2024-03-29T10:38:53.696 INFO:teuthology.orchestra.run.smithi179.stdout: osd scrub max interval = 600 2024-03-29T10:38:53.696 INFO:teuthology.orchestra.run.smithi179.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:38:53.696 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.696 INFO:teuthology.orchestra.run.smithi179.stdout: osd recover clone overlap = true 2024-03-29T10:38:53.696 INFO:teuthology.orchestra.run.smithi179.stdout: osd recovery max chunk = 1048576 2024-03-29T10:38:53.696 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.696 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug shutdown = true 2024-03-29T10:38:53.696 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug op order = true 2024-03-29T10:38:53.697 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug verify stray on activate = true 2024-03-29T10:38:53.697 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug trim objects = true 2024-03-29T10:38:53.697 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.697 INFO:teuthology.orchestra.run.smithi179.stdout: osd open classes on start = true 2024-03-29T10:38:53.697 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug pg log writeout = true 2024-03-29T10:38:53.697 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.697 INFO:teuthology.orchestra.run.smithi179.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:38:53.697 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.697 INFO:teuthology.orchestra.run.smithi179.stdout: osd map max advance = 10 2024-03-29T10:38:53.697 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.697 INFO:teuthology.orchestra.run.smithi179.stdout: journal zero on create = true 2024-03-29T10:38:53.697 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.697 INFO:teuthology.orchestra.run.smithi179.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:38:53.697 INFO:teuthology.orchestra.run.smithi179.stdout: filestore apply finisher threads = 3 2024-03-29T10:38:53.698 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.698 INFO:teuthology.orchestra.run.smithi179.stdout: bdev debug aio = true 2024-03-29T10:38:53.698 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug misdirected ops = true 2024-03-29T10:38:53.698 INFO:teuthology.orchestra.run.smithi179.stdout: bluestore block size = 96636764160 2024-03-29T10:38:53.698 INFO:teuthology.orchestra.run.smithi179.stdout: bluestore compression algorithm = zstd 2024-03-29T10:38:53.698 INFO:teuthology.orchestra.run.smithi179.stdout: bluestore compression mode = aggressive 2024-03-29T10:38:53.698 INFO:teuthology.orchestra.run.smithi179.stdout: bluestore fsck on mount = True 2024-03-29T10:38:53.698 INFO:teuthology.orchestra.run.smithi179.stdout: bluestore zero block detection = True 2024-03-29T10:38:53.698 INFO:teuthology.orchestra.run.smithi179.stdout: debug bluefs = 20 2024-03-29T10:38:53.698 INFO:teuthology.orchestra.run.smithi179.stdout: debug bluestore = 20 2024-03-29T10:38:53.698 INFO:teuthology.orchestra.run.smithi179.stdout: debug monc = 20 2024-03-29T10:38:53.698 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2024-03-29T10:38:53.698 INFO:teuthology.orchestra.run.smithi179.stdout: debug osd = 20 2024-03-29T10:38:53.698 INFO:teuthology.orchestra.run.smithi179.stdout: debug rocksdb = 10 2024-03-29T10:38:53.698 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:38:53.699 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd full ratio = 0.9 2024-03-29T10:38:53.699 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:38:53.699 INFO:teuthology.orchestra.run.smithi179.stdout: osd class default list = * 2024-03-29T10:38:53.699 INFO:teuthology.orchestra.run.smithi179.stdout: osd class load list = * 2024-03-29T10:38:53.699 INFO:teuthology.orchestra.run.smithi179.stdout: osd client watch timeout = 120 2024-03-29T10:38:53.699 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug verify cached snaps = True 2024-03-29T10:38:53.699 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug verify missing on start = True 2024-03-29T10:38:53.699 INFO:teuthology.orchestra.run.smithi179.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:38:53.699 INFO:teuthology.orchestra.run.smithi179.stdout: osd mclock override recovery settings = True 2024-03-29T10:38:53.699 INFO:teuthology.orchestra.run.smithi179.stdout: osd objectstore = bluestore 2024-03-29T10:38:53.699 INFO:teuthology.orchestra.run.smithi179.stdout: osd op queue = debug_random 2024-03-29T10:38:53.699 INFO:teuthology.orchestra.run.smithi179.stdout: osd op queue cut off = debug_random 2024-03-29T10:38:53.699 INFO:teuthology.orchestra.run.smithi179.stdout: osd shutdown pgref assert = True 2024-03-29T10:38:53.699 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.699 INFO:teuthology.orchestra.run.smithi179.stdout:[mgr] 2024-03-29T10:38:53.700 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2024-03-29T10:38:53.700 INFO:teuthology.orchestra.run.smithi179.stdout: debug mgr = 20 2024-03-29T10:38:53.700 INFO:teuthology.orchestra.run.smithi179.stdout: debug mon = 20 2024-03-29T10:38:53.700 INFO:teuthology.orchestra.run.smithi179.stdout: debug auth = 20 2024-03-29T10:38:53.700 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:38:53.700 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:38:53.700 INFO:teuthology.orchestra.run.smithi179.stdout: mgr/telemetry/nag = false 2024-03-29T10:38:53.700 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.700 INFO:teuthology.orchestra.run.smithi179.stdout:[mon] 2024-03-29T10:38:53.700 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2024-03-29T10:38:53.700 INFO:teuthology.orchestra.run.smithi179.stdout: debug mon = 20 2024-03-29T10:38:53.700 INFO:teuthology.orchestra.run.smithi179.stdout: debug paxos = 20 2024-03-29T10:38:53.700 INFO:teuthology.orchestra.run.smithi179.stdout: debug auth = 20 2024-03-29T10:38:53.700 INFO:teuthology.orchestra.run.smithi179.stdout: mon data avail warn = 5 2024-03-29T10:38:53.700 INFO:teuthology.orchestra.run.smithi179.stdout: mon mgr mkfs grace = 240 2024-03-29T10:38:53.701 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:38:53.701 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd reporter subtree level = osd 2024-03-29T10:38:53.701 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd prime pg temp = true 2024-03-29T10:38:53.701 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:38:53.701 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.701 INFO:teuthology.orchestra.run.smithi179.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:38:53.701 INFO:teuthology.orchestra.run.smithi179.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:38:53.701 INFO:teuthology.orchestra.run.smithi179.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:38:53.701 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.701 INFO:teuthology.orchestra.run.smithi179.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:38:53.701 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:38:53.701 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:38:53.701 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.701 INFO:teuthology.orchestra.run.smithi179.stdout: # 1m isn't quite enough 2024-03-29T10:38:53.701 INFO:teuthology.orchestra.run.smithi179.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:38:53.702 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.702 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:38:53.702 INFO:teuthology.orchestra.run.smithi179.stdout: mon min osdmap epochs = 50 2024-03-29T10:38:53.702 INFO:teuthology.orchestra.run.smithi179.stdout: mon osdmap full prune interval = 2 2024-03-29T10:38:53.702 INFO:teuthology.orchestra.run.smithi179.stdout: mon osdmap full prune min = 15 2024-03-29T10:38:53.702 INFO:teuthology.orchestra.run.smithi179.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:38:53.702 INFO:teuthology.orchestra.run.smithi179.stdout: mon scrub interval = 300 2024-03-29T10:38:53.702 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on pool no app = False 2024-03-29T10:38:53.702 INFO:teuthology.orchestra.run.smithi179.stdout: paxos service trim min = 10 2024-03-29T10:38:53.702 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-29T10:38:53.702 INFO:teuthology.orchestra.run.smithi179.stdout:[client] 2024-03-29T10:38:53.702 INFO:teuthology.orchestra.run.smithi179.stdout: rgw cache enabled = true 2024-03-29T10:38:53.702 INFO:teuthology.orchestra.run.smithi179.stdout: rgw enable ops log = true 2024-03-29T10:38:53.702 INFO:teuthology.orchestra.run.smithi179.stdout: rgw enable usage log = true 2024-03-29T10:38:53.703 INFO:teuthology.orchestra.run.smithi179.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:38:53.703 INFO:teuthology.orchestra.run.smithi179.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:38:53.703 INFO:teuthology.orchestra.run.smithi179.stdout: debug monc = 20 2024-03-29T10:38:53.703 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2024-03-29T10:38:53.703 INFO:teuthology.orchestra.run.smithi179.stdout: debug objecter = 20 2024-03-29T10:38:53.703 INFO:teuthology.orchestra.run.smithi179.stdout: debug rados = 20 2024-03-29T10:38:53.703 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.a] 2024-03-29T10:38:53.703 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.c] 2024-03-29T10:38:53.703 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.b] 2024-03-29T10:38:53.704 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T10:38:53.705 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T10:38:53.760 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T10:38:53.760 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-29T10:38:53.760 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T10:38:53.774 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-29T10:38:53.774 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T10:38:53.832 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi057.front.sepia.ceph.com 2024-03-29T10:38:53.832 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-29T10:38:53.832 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:38:53.832 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:38:53.913 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-29T10:38:53.913 DEBUG:teuthology.orchestra.run.smithi057:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:53.967 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi179.front.sepia.ceph.com 2024-03-29T10:38:53.967 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T10:38:53.967 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:38:53.967 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:38:54.007 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T10:38:54.007 DEBUG:teuthology.orchestra.run.smithi179:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:38:54.062 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T10:38:54.062 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T10:38:54.062 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:38:54.116 INFO:teuthology.orchestra.run.smithi057.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:38:54.119 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:38:54.182 INFO:teuthology.orchestra.run.smithi179.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:38:54.184 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T10:38:54.185 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T10:38:54.185 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T10:38:54.227 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T10:38:54.242 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T10:38:54.284 INFO:teuthology.orchestra.run.smithi179.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T10:38:54.299 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T10:38:54.299 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi057.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi179.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:38:54.299 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T10:38:54.326 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:54.326 INFO:tasks.ceph:role: osd.0 2024-03-29T10:38:54.326 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi057.front.sepia.ceph.com 2024-03-29T10:38:54.326 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:38:54.422 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:38:54.422 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:38:54.422 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:38:54.422 INFO:teuthology.orchestra.run.smithi057.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:38:54.422 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:38:54.422 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:38:54.422 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:38:54.422 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:38:54.422 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:38:54.422 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:38:55.779 INFO:teuthology.orchestra.run.smithi057.stdout:Discarding blocks...Done. 2024-03-29T10:38:55.781 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2024-03-29T10:38:55.782 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T10:38:55.829 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T10:38:55.863 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T10:38:55.932 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:55.932 INFO:tasks.ceph:role: osd.1 2024-03-29T10:38:55.932 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi057.front.sepia.ceph.com 2024-03-29T10:38:55.933 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:38:56.028 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:38:56.028 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:38:56.028 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:38:56.029 INFO:teuthology.orchestra.run.smithi057.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:38:56.029 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:38:56.029 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:38:56.029 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:38:56.029 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:38:56.029 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:38:56.029 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:38:57.392 INFO:teuthology.orchestra.run.smithi057.stdout:Discarding blocks...Done. 2024-03-29T10:38:57.394 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2024-03-29T10:38:57.394 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T10:38:57.428 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T10:38:57.499 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T10:38:57.566 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:57.566 INFO:tasks.ceph:role: osd.2 2024-03-29T10:38:57.566 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi057.front.sepia.ceph.com 2024-03-29T10:38:57.566 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:38:57.660 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:38:57.661 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:38:57.661 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:38:57.661 INFO:teuthology.orchestra.run.smithi057.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:38:57.661 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:38:57.661 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:38:57.661 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:38:57.661 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:38:57.661 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:38:57.661 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:38:58.980 INFO:teuthology.orchestra.run.smithi057.stdout:Discarding blocks...Done. 2024-03-29T10:38:58.983 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2024-03-29T10:38:58.983 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T10:38:59.020 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T10:38:59.094 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T10:38:59.162 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:38:59.162 INFO:tasks.ceph:role: osd.3 2024-03-29T10:38:59.163 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi057.front.sepia.ceph.com 2024-03-29T10:38:59.163 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:38:59.258 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:38:59.259 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:38:59.259 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:38:59.259 INFO:teuthology.orchestra.run.smithi057.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:38:59.259 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:38:59.259 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:38:59.259 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:38:59.259 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:38:59.259 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:38:59.259 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:39:00.613 INFO:teuthology.orchestra.run.smithi057.stdout:Discarding blocks...Done. 2024-03-29T10:39:00.615 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2024-03-29T10:39:00.616 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T10:39:00.655 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T10:39:00.733 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:39:00.840 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:00.856+0000 7fd43ed1f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T10:39:00.841 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:00.857+0000 7fd43ed1f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T10:39:00.841 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:00.857+0000 7fd43ed1f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:39:00.841 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:00.857+0000 7fd43ed1f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:39:00.841 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:00.857+0000 7fd43ed1f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:39:00.841 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:00.858+0000 7fd43ed1f640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T10:39:03.710 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T10:39:03.741 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:39:03.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:03.849+0000 7fb1bfb8f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T10:39:03.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:03.850+0000 7fb1bfb8f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T10:39:03.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:03.850+0000 7fb1bfb8f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:39:03.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:03.850+0000 7fb1bfb8f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:39:03.834 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:03.850+0000 7fb1bfb8f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:39:03.835 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:03.851+0000 7fb1bfb8f640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T10:39:06.705 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T10:39:06.737 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:39:06.828 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:06.844+0000 7fa021813640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T10:39:06.828 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:06.844+0000 7fa021813640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T10:39:06.829 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:06.844+0000 7fa021813640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:39:06.829 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:06.844+0000 7fa021813640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:39:06.829 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:06.844+0000 7fa021813640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:39:06.829 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:06.845+0000 7fa021813640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T10:39:09.711 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T10:39:09.742 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:39:09.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:09.848+0000 7f8ab9613640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T10:39:09.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:09.848+0000 7f8ab9613640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T10:39:09.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:09.848+0000 7f8ab9613640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:39:09.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:09.848+0000 7f8ab9613640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:39:09.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:09.848+0000 7f8ab9613640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:39:09.833 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:09.849+0000 7f8ab9613640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T10:39:12.700 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T10:39:12.732 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T10:39:12.766 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:39:12.766 INFO:tasks.ceph:role: osd.4 2024-03-29T10:39:12.766 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi179.front.sepia.ceph.com 2024-03-29T10:39:12.766 DEBUG:teuthology.orchestra.run.smithi179:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:39:12.846 INFO:teuthology.orchestra.run.smithi179.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:39:12.846 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:39:12.846 INFO:teuthology.orchestra.run.smithi179.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:39:12.846 INFO:teuthology.orchestra.run.smithi179.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:39:12.847 INFO:teuthology.orchestra.run.smithi179.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:39:12.847 INFO:teuthology.orchestra.run.smithi179.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:39:12.847 INFO:teuthology.orchestra.run.smithi179.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:39:12.847 INFO:teuthology.orchestra.run.smithi179.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:39:12.847 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:39:12.847 INFO:teuthology.orchestra.run.smithi179.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:39:13.349 INFO:teuthology.orchestra.run.smithi179.stdout:Discarding blocks...Done. 2024-03-29T10:39:13.352 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi179.front.sepia.ceph.com -o noatime 2024-03-29T10:39:13.352 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T10:39:13.509 DEBUG:teuthology.orchestra.run.smithi179:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T10:39:13.546 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T10:39:13.615 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:39:13.615 INFO:tasks.ceph:role: osd.5 2024-03-29T10:39:13.615 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi179.front.sepia.ceph.com 2024-03-29T10:39:13.616 DEBUG:teuthology.orchestra.run.smithi179:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:39:13.690 INFO:teuthology.orchestra.run.smithi179.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:39:13.691 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:39:13.691 INFO:teuthology.orchestra.run.smithi179.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:39:13.691 INFO:teuthology.orchestra.run.smithi179.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:39:13.691 INFO:teuthology.orchestra.run.smithi179.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:39:13.691 INFO:teuthology.orchestra.run.smithi179.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:39:13.691 INFO:teuthology.orchestra.run.smithi179.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:39:13.691 INFO:teuthology.orchestra.run.smithi179.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:39:13.691 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:39:13.691 INFO:teuthology.orchestra.run.smithi179.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:39:14.132 INFO:teuthology.orchestra.run.smithi179.stdout:Discarding blocks...Done. 2024-03-29T10:39:14.134 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi179.front.sepia.ceph.com -o noatime 2024-03-29T10:39:14.134 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T10:39:14.229 DEBUG:teuthology.orchestra.run.smithi179:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T10:39:14.260 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T10:39:14.327 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:39:14.327 INFO:tasks.ceph:role: osd.6 2024-03-29T10:39:14.327 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi179.front.sepia.ceph.com 2024-03-29T10:39:14.327 DEBUG:teuthology.orchestra.run.smithi179:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:39:14.401 INFO:teuthology.orchestra.run.smithi179.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:39:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:39:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:39:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:39:14.401 INFO:teuthology.orchestra.run.smithi179.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:39:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:39:14.401 INFO:teuthology.orchestra.run.smithi179.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:39:14.401 INFO:teuthology.orchestra.run.smithi179.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:39:14.401 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:39:14.401 INFO:teuthology.orchestra.run.smithi179.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:39:14.880 INFO:teuthology.orchestra.run.smithi179.stdout:Discarding blocks...Done. 2024-03-29T10:39:14.882 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi179.front.sepia.ceph.com -o noatime 2024-03-29T10:39:14.882 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T10:39:15.036 DEBUG:teuthology.orchestra.run.smithi179:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T10:39:15.068 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T10:39:15.136 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:39:15.136 INFO:tasks.ceph:role: osd.7 2024-03-29T10:39:15.136 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi179.front.sepia.ceph.com 2024-03-29T10:39:15.136 DEBUG:teuthology.orchestra.run.smithi179:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:39:15.211 INFO:teuthology.orchestra.run.smithi179.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:39:15.211 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:39:15.212 INFO:teuthology.orchestra.run.smithi179.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:39:15.212 INFO:teuthology.orchestra.run.smithi179.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:39:15.212 INFO:teuthology.orchestra.run.smithi179.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:39:15.212 INFO:teuthology.orchestra.run.smithi179.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:39:15.212 INFO:teuthology.orchestra.run.smithi179.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:39:15.212 INFO:teuthology.orchestra.run.smithi179.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:39:15.212 INFO:teuthology.orchestra.run.smithi179.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:39:15.212 INFO:teuthology.orchestra.run.smithi179.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:39:15.737 INFO:teuthology.orchestra.run.smithi179.stdout:Discarding blocks...Done. 2024-03-29T10:39:15.739 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi179.front.sepia.ceph.com -o noatime 2024-03-29T10:39:15.739 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T10:39:15.882 DEBUG:teuthology.orchestra.run.smithi179:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T10:39:15.917 DEBUG:teuthology.orchestra.run.smithi179:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:39:16.014 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:16.029+0000 7fc9894c2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T10:39:16.014 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:16.030+0000 7fc9894c2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T10:39:16.014 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:16.030+0000 7fc9894c2640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:39:16.014 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:16.030+0000 7fc9894c2640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:39:16.014 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:16.030+0000 7fc9894c2640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:39:16.015 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:16.030+0000 7fc9894c2640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T10:39:18.864 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T10:39:18.895 DEBUG:teuthology.orchestra.run.smithi179:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:39:18.987 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:19.003+0000 7ffbf5423640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T10:39:18.988 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:19.004+0000 7ffbf5423640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T10:39:18.988 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:19.004+0000 7ffbf5423640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:39:18.988 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:19.004+0000 7ffbf5423640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:39:18.988 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:19.004+0000 7ffbf5423640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:39:18.988 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:19.004+0000 7ffbf5423640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T10:39:21.832 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T10:39:21.863 DEBUG:teuthology.orchestra.run.smithi179:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:39:21.956 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:21.971+0000 7f09c8aa6640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T10:39:21.956 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:21.971+0000 7f09c8aa6640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T10:39:21.956 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:21.971+0000 7f09c8aa6640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:39:21.956 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:21.971+0000 7f09c8aa6640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:39:21.957 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:21.971+0000 7f09c8aa6640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:39:21.957 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:21.972+0000 7f09c8aa6640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T10:39:24.808 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T10:39:24.840 DEBUG:teuthology.orchestra.run.smithi179:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:39:24.930 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:24.946+0000 7f3577c48640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T10:39:24.930 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:24.946+0000 7f3577c48640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T10:39:24.931 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:24.946+0000 7f3577c48640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:39:24.931 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:24.946+0000 7f3577c48640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:39:24.931 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:24.946+0000 7f3577c48640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:39:24.931 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:24.947+0000 7f3577c48640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T10:39:27.780 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T10:39:27.808 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T10:39:27.809 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-29T10:39:27.809 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T10:39:27.841 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-29T10:39:27.841 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T10:39:27.912 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-29T10:39:27.912 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T10:39:27.982 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-29T10:39:27.982 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T10:39:28.050 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-29T10:39:28.050 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T10:39:28.118 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T10:39:28.119 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T10:39:28.148 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T10:39:28.148 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T10:39:28.217 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T10:39:28.218 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T10:39:28.285 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T10:39:28.286 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T10:39:28.353 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T10:39:28.353 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T10:39:28.421 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-29T10:39:28.422 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T10:39:28.438 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T10:39:28.438 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T10:39:28.477 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T10:39:28.477 DEBUG:teuthology.orchestra.run.smithi057:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:39:28.481 DEBUG:teuthology.orchestra.run.smithi179:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:39:28.520 INFO:teuthology.orchestra.run.smithi057.stdout:[mgr.y] 2024-03-29T10:39:28.520 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQA+mgZm0Oz3BxAAnIK5t/dgTUlk+4GtN5TT4A== 2024-03-29T10:39:28.520 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.0] 2024-03-29T10:39:28.520 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBEmgZmwqAbMxAAbNLDwfpARBuQFpPN28CkgQ== 2024-03-29T10:39:28.520 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.1] 2024-03-29T10:39:28.520 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBHmgZmcbSxMhAAyGsgBBw7mKUdgL4hIXSxvw== 2024-03-29T10:39:28.520 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.2] 2024-03-29T10:39:28.520 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBKmgZmqdZgMhAAtqheHgr14xikWHrdM7geIw== 2024-03-29T10:39:28.520 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.3] 2024-03-29T10:39:28.520 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBNmgZmp8GYMhAAFObVYmBHqPRPgHE1FSUYcw== 2024-03-29T10:39:28.521 INFO:teuthology.orchestra.run.smithi057.stdout:[mgr.x] 2024-03-29T10:39:28.521 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQA+mgZmK4zeCxAAHrrJAeqNEMoOu3csUkRnnA== 2024-03-29T10:39:28.521 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.4] 2024-03-29T10:39:28.521 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBUmgZm6irXARAAmOR/h8PPpEiAiN1RCaZ0WA== 2024-03-29T10:39:28.521 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.5] 2024-03-29T10:39:28.521 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBXmgZm38xGABAAPhyKrkmjYl6T1v7WHCCE9g== 2024-03-29T10:39:28.521 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.6] 2024-03-29T10:39:28.521 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBZmgZmraP0ORAAODAt2+SHTNNgG1hM8xSPYw== 2024-03-29T10:39:28.521 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.7] 2024-03-29T10:39:28.521 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQBcmgZmS1l0OBAAEyrsEj0GTN7ranotE1JMew== 2024-03-29T10:39:28.521 INFO:teuthology.orchestra.run.smithi057.stdout:[client.0] 2024-03-29T10:39:28.521 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQA+mgZmEEyVDhAABYIuJRtjhijQiqmer45oOg== 2024-03-29T10:39:28.521 INFO:teuthology.orchestra.run.smithi057.stdout:[client.1] 2024-03-29T10:39:28.522 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQA+mgZmaYH6ERAAfQkjW2BRrgqLLjOeSjDVhg== 2024-03-29T10:39:28.542 INFO:teuthology.orchestra.run.smithi179.stdout:[mgr.y] 2024-03-29T10:39:28.542 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQA+mgZm0Oz3BxAAnIK5t/dgTUlk+4GtN5TT4A== 2024-03-29T10:39:28.542 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.0] 2024-03-29T10:39:28.542 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQBEmgZmwqAbMxAAbNLDwfpARBuQFpPN28CkgQ== 2024-03-29T10:39:28.542 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.1] 2024-03-29T10:39:28.542 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQBHmgZmcbSxMhAAyGsgBBw7mKUdgL4hIXSxvw== 2024-03-29T10:39:28.542 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.2] 2024-03-29T10:39:28.542 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQBKmgZmqdZgMhAAtqheHgr14xikWHrdM7geIw== 2024-03-29T10:39:28.542 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.3] 2024-03-29T10:39:28.542 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQBNmgZmp8GYMhAAFObVYmBHqPRPgHE1FSUYcw== 2024-03-29T10:39:28.543 INFO:teuthology.orchestra.run.smithi179.stdout:[mgr.x] 2024-03-29T10:39:28.543 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQA+mgZmK4zeCxAAHrrJAeqNEMoOu3csUkRnnA== 2024-03-29T10:39:28.543 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.4] 2024-03-29T10:39:28.543 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQBUmgZm6irXARAAmOR/h8PPpEiAiN1RCaZ0WA== 2024-03-29T10:39:28.543 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.5] 2024-03-29T10:39:28.543 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQBXmgZm38xGABAAPhyKrkmjYl6T1v7WHCCE9g== 2024-03-29T10:39:28.543 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.6] 2024-03-29T10:39:28.543 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQBZmgZmraP0ORAAODAt2+SHTNNgG1hM8xSPYw== 2024-03-29T10:39:28.543 INFO:teuthology.orchestra.run.smithi179.stdout:[osd.7] 2024-03-29T10:39:28.543 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQBcmgZmS1l0OBAAEyrsEj0GTN7ranotE1JMew== 2024-03-29T10:39:28.543 INFO:teuthology.orchestra.run.smithi179.stdout:[client.0] 2024-03-29T10:39:28.543 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQA+mgZmEEyVDhAABYIuJRtjhijQiqmer45oOg== 2024-03-29T10:39:28.543 INFO:teuthology.orchestra.run.smithi179.stdout:[client.1] 2024-03-29T10:39:28.543 INFO:teuthology.orchestra.run.smithi179.stdout: key = AQA+mgZmaYH6ERAAfQkjW2BRrgqLLjOeSjDVhg== 2024-03-29T10:39:28.544 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:39:28.565 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:39:28.636 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:28.638 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:28.690 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:28.692 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:28.743 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:28.746 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:28.797 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:28.799 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:28.851 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:39:28.853 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:39:28.905 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:28.907 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:28.958 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:28.960 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:29.012 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:29.014 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:29.066 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:29.068 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:39:29.119 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:39:29.122 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:39:29.177 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:39:29.180 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:39:29.235 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T10:39:29.236 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T10:39:29.261 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:39:29.361 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T10:39:29.388 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T10:39:29.455 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:39:29.565 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T10:39:29.592 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T10:39:29.619 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:39:29.716 DEBUG:teuthology.orchestra.run.smithi179:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T10:39:29.743 DEBUG:teuthology.orchestra.run.smithi057:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:39:29.745 DEBUG:teuthology.orchestra.run.smithi179:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:39:29.797 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T10:39:29.797 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T10:39:29.798 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T10:39:29.800 INFO:tasks.ceph.mon.a:Started 2024-03-29T10:39:29.800 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T10:39:29.800 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T10:39:29.802 INFO:tasks.ceph.mon.c:Started 2024-03-29T10:39:29.802 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T10:39:29.802 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T10:39:29.840 INFO:tasks.ceph.mon.b:Started 2024-03-29T10:39:29.840 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T10:39:29.840 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T10:39:29.840 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T10:39:29.842 INFO:tasks.ceph.mgr.y:Started 2024-03-29T10:39:29.843 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T10:39:29.843 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T10:39:29.845 INFO:tasks.ceph.mgr.x:Started 2024-03-29T10:39:29.845 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T10:39:29.845 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T10:39:29.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:29.963+0000 7fb182f5d640 10 monclient: get_monmap_and_config 2024-03-29T10:39:29.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:29.963+0000 7fb182f5d640 10 monclient: build_initial_monmap 2024-03-29T10:39:29.948 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:29.963+0000 7fb182f5d640 10 monclient: monmap: 2024-03-29T10:39:29.948 INFO:teuthology.orchestra.run.smithi057.stderr:epoch 0 2024-03-29T10:39:29.948 INFO:teuthology.orchestra.run.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:29.948 INFO:teuthology.orchestra.run.smithi057.stderr:last_changed 2024-03-29T10:39:29.964594+0000 2024-03-29T10:39:29.948 INFO:teuthology.orchestra.run.smithi057.stderr:created 2024-03-29T10:39:29.964594+0000 2024-03-29T10:39:29.948 INFO:teuthology.orchestra.run.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:29.948 INFO:teuthology.orchestra.run.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:29.948 INFO:teuthology.orchestra.run.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:29.948 INFO:teuthology.orchestra.run.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:39:29.949 INFO:teuthology.orchestra.run.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:39:29.949 INFO:teuthology.orchestra.run.smithi057.stderr: 2024-03-29T10:39:29.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:29.964+0000 7fb182f5d640 1 Processor -- start 2024-03-29T10:39:29.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:29.964+0000 7fb182f5d640 1 -- start start 2024-03-29T10:39:29.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:29.964+0000 7fb182f5d640 10 monclient: init 2024-03-29T10:39:29.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:29.965+0000 7fb182f5d640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:29.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:29.965+0000 7fb182f5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:39:29.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:29.965+0000 7fb182f5d640 10 monclient(hunting): picked mon.noname-b con 0x7fb17c09ac40 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:29.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:29.965+0000 7fb182f5d640 10 monclient(hunting): picked mon.noname-c con 0x7fb17c09c870 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:29.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:29.965+0000 7fb182f5d640 10 monclient(hunting): picked mon.noname-a con 0x7fb17c0b4360 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:29.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:29.965+0000 7fb182f5d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb17c09dec0 con 0x7fb17c0b4360 2024-03-29T10:39:29.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:29.965+0000 7fb182f5d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb17c09da30 con 0x7fb17c09c870 2024-03-29T10:39:29.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:29.965+0000 7fb182f5d640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb17c0b39c0 con 0x7fb17c09ac40 2024-03-29T10:39:29.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:29.965+0000 7fb182f5d640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:29.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:29.965+0000 7fb182f5d640 10 monclient(hunting): authenticate will time out at 922.141268s 2024-03-29T10:39:29.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:29.965+0000 7fb1814fa640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fb17c0b4360 0x7fb17c14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:58250/0 (socket says 172.21.15.57:58250) 2024-03-29T10:39:29.951 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:29.965+0000 7fb1814fa640 1 -- 172.21.15.57:0/2483893324 learned_addr learned my addr 172.21.15.57:0/2483893324 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:39:32.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:32.965+0000 7fb17a7fc640 10 monclient(hunting): tick 2024-03-29T10:39:32.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:32.965+0000 7fb17a7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-29T10:39:32.949 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:32.965+0000 7fb17a7fc640 1 monclient(hunting): continuing hunt 2024-03-29T10:39:32.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:32.965+0000 7fb17a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T10:39:32.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:32.965+0000 7fb17a7fc640 1 -- 172.21.15.57:0/2483893324 >> v1:172.21.15.57:6790/0 conn(0x7fb17c09ac40 legacy=0x7fb17c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:32.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:32.965+0000 7fb17a7fc640 1 -- 172.21.15.57:0/2483893324 >> v1:172.21.15.179:6789/0 conn(0x7fb17c09c870 legacy=0x7fb17c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:32.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:32.965+0000 7fb17a7fc640 1 -- 172.21.15.57:0/2483893324 >> v1:172.21.15.57:6789/0 conn(0x7fb17c0b4360 legacy=0x7fb17c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:32.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:32.965+0000 7fb17a7fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:39:32.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:32.965+0000 7fb17a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7fb1600036e0 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:32.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:32.965+0000 7fb17a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7fb160007300 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:32.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:32.965+0000 7fb17a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7fb16000b020 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:32.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:32.965+0000 7fb17a7fc640 1 -- 172.21.15.57:0/2483893324 --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb17c0b5160 con 0x7fb1600036e0 2024-03-29T10:39:32.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:32.965+0000 7fb17a7fc640 1 -- 172.21.15.57:0/2483893324 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb16000fcd0 con 0x7fb16000b020 2024-03-29T10:39:32.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:32.965+0000 7fb17a7fc640 1 -- 172.21.15.57:0/2483893324 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb160010f10 con 0x7fb160007300 2024-03-29T10:39:32.950 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:32.965+0000 7fb17a7fc640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:35.179 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.195+0000 7fb17b7fe640 1 -- 172.21.15.57:0/2483893324 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023827998 0 0) 0x7fb17c0b5160 con 0x7fb1600036e0 2024-03-29T10:39:35.179 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.195+0000 7fb17b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:35.179 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.195+0000 7fb17b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:35.179 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.195+0000 7fb17b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:35.179 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.195+0000 7fb17b7fe640 10 monclient(hunting): my global_id is 4101 2024-03-29T10:39:35.179 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.195+0000 7fb17b7fe640 1 -- 172.21.15.57:0/2483893324 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb16c003520 con 0x7fb1600036e0 2024-03-29T10:39:35.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.195+0000 7fb17b7fe640 1 -- 172.21.15.57:0/2483893324 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434941601 0 0) 0x7fb160010f10 con 0x7fb160007300 2024-03-29T10:39:35.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.195+0000 7fb17b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:35.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.195+0000 7fb17b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:35.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.195+0000 7fb17b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:35.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.195+0000 7fb17b7fe640 10 monclient(hunting): my global_id is 4106 2024-03-29T10:39:35.180 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.195+0000 7fb17b7fe640 1 -- 172.21.15.57:0/2483893324 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb17c0b5160 con 0x7fb160007300 2024-03-29T10:39:35.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb17b7fe640 1 -- 172.21.15.57:0/2483893324 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3967910333 0 0) 0x7fb17c0b5160 con 0x7fb160007300 2024-03-29T10:39:35.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb17b7fe640 1 -- 172.21.15.57:0/2483893324 >> v1:172.21.15.179:6789/0 conn(0x7fb16000b020 legacy=0x7fb16000d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:35.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb17b7fe640 1 -- 172.21.15.57:0/2483893324 >> v1:172.21.15.57:6789/0 conn(0x7fb1600036e0 legacy=0x7fb160005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:35.182 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb17b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:39:35.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb17b7fe640 1 monclient: found mon.noname-b 2024-03-29T10:39:35.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb17b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:35.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb17b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:39:35.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb17b7fe640 1 -- 172.21.15.57:0/2483893324 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb160012170 con 0x7fb160007300 2024-03-29T10:39:35.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb17b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:39:35.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb17b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:39:35.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb17b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:35.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb17b7fe640 1 -- 172.21.15.57:0/2483893324 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb168001610 con 0x7fb160007300 2024-03-29T10:39:35.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb17b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:35.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb17b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:35.183 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb17b7fe640 10 monclient: dump: 2024-03-29T10:39:35.183 INFO:teuthology.orchestra.run.smithi057.stderr:epoch 1 2024-03-29T10:39:35.183 INFO:teuthology.orchestra.run.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:35.183 INFO:teuthology.orchestra.run.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:35.184 INFO:teuthology.orchestra.run.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:35.184 INFO:teuthology.orchestra.run.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:35.184 INFO:teuthology.orchestra.run.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:35.184 INFO:teuthology.orchestra.run.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:35.184 INFO:teuthology.orchestra.run.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:35.184 INFO:teuthology.orchestra.run.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:35.184 INFO:teuthology.orchestra.run.smithi057.stderr: 2024-03-29T10:39:35.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb182f5d640 5 monclient: authenticate success, global_id 4106 2024-03-29T10:39:35.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb182f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:35.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb17b7fe640 1 -- 172.21.15.57:0/2483893324 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb168001770 con 0x7fb160007300 2024-03-29T10:39:35.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb17b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:35.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb17b7fe640 1 -- 172.21.15.57:0/2483893324 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb168001bc0 con 0x7fb160007300 2024-03-29T10:39:35.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb182f5d640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:35.184 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb182f5d640 4 set_mon_vals no callback set 2024-03-29T10:39:35.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb17b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:35.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb17b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:35.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.196+0000 7fb17b7fe640 10 monclient: dump: 2024-03-29T10:39:35.185 INFO:teuthology.orchestra.run.smithi057.stderr:epoch 1 2024-03-29T10:39:35.185 INFO:teuthology.orchestra.run.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:35.185 INFO:teuthology.orchestra.run.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:35.185 INFO:teuthology.orchestra.run.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:35.185 INFO:teuthology.orchestra.run.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:35.185 INFO:teuthology.orchestra.run.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:35.185 INFO:teuthology.orchestra.run.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:35.185 INFO:teuthology.orchestra.run.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:35.185 INFO:teuthology.orchestra.run.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:35.185 INFO:teuthology.orchestra.run.smithi057.stderr: 2024-03-29T10:39:35.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.197+0000 7fb182f5d640 10 monclient: shutdown 2024-03-29T10:39:35.185 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.197+0000 7fb182f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:35.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.197+0000 7fb182f5d640 1 -- 172.21.15.57:0/2483893324 >> v1:172.21.15.57:6790/0 conn(0x7fb160007300 legacy=0x7fb160009790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:35.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.197+0000 7fb182f5d640 1 -- 172.21.15.57:0/2483893324 shutdown_connections 2024-03-29T10:39:35.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.197+0000 7fb182f5d640 1 -- 172.21.15.57:0/2483893324 >> 172.21.15.57:0/2483893324 conn(0x7fb17c066b00 msgr2=0x7fb17c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:35.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.197+0000 7fb182f5d640 1 -- 172.21.15.57:0/2483893324 shutdown_connections 2024-03-29T10:39:35.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.197+0000 7fb182f5d640 1 -- 172.21.15.57:0/2483893324 wait complete. 2024-03-29T10:39:35.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.198+0000 7fb182f5d640 10 monclient: build_initial_monmap 2024-03-29T10:39:35.186 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.198+0000 7fb182f5d640 10 monclient: monmap: 2024-03-29T10:39:35.186 INFO:teuthology.orchestra.run.smithi057.stderr:epoch 0 2024-03-29T10:39:35.186 INFO:teuthology.orchestra.run.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:35.186 INFO:teuthology.orchestra.run.smithi057.stderr:last_changed 0.000000 2024-03-29T10:39:35.186 INFO:teuthology.orchestra.run.smithi057.stderr:created 0.000000 2024-03-29T10:39:35.186 INFO:teuthology.orchestra.run.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:35.186 INFO:teuthology.orchestra.run.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:35.186 INFO:teuthology.orchestra.run.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:35.186 INFO:teuthology.orchestra.run.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:39:35.187 INFO:teuthology.orchestra.run.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:39:35.187 INFO:teuthology.orchestra.run.smithi057.stderr: 2024-03-29T10:39:35.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.198+0000 7fb182f5d640 1 librados: starting msgr at 2024-03-29T10:39:35.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.198+0000 7fb182f5d640 1 librados: starting objecter 2024-03-29T10:39:35.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.199+0000 7fb182f5d640 1 Processor -- start 2024-03-29T10:39:35.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.199+0000 7fb182f5d640 1 -- start start 2024-03-29T10:39:35.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.199+0000 7fb182f5d640 1 librados: setting wanted keys 2024-03-29T10:39:35.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.199+0000 7fb182f5d640 1 librados: calling monclient init 2024-03-29T10:39:35.187 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.199+0000 7fb182f5d640 10 monclient: init 2024-03-29T10:39:35.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.200+0000 7fb182f5d640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:35.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.200+0000 7fb182f5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:35.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.200+0000 7fb182f5d640 10 monclient(hunting): picked mon.noname-b con 0x7fb1600036e0 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:35.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.200+0000 7fb182f5d640 10 monclient(hunting): picked mon.noname-c con 0x7fb160007300 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:35.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.200+0000 7fb182f5d640 10 monclient(hunting): picked mon.noname-a con 0x7fb16000b020 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:35.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.200+0000 7fb182f5d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb17c139900 con 0x7fb16000b020 2024-03-29T10:39:35.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.200+0000 7fb1814fa640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fb16000b020 0x7fb17c1391f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46164/0 (socket says 172.21.15.57:46164) 2024-03-29T10:39:35.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.200+0000 7fb17bfff640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7fb160007300 0x7fb17c138ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:40958/0 (socket says 172.21.15.57:40958) 2024-03-29T10:39:35.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.200+0000 7fb1814fa640 1 -- 172.21.15.57:0/3914989047 learned_addr learned my addr 172.21.15.57:0/3914989047 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:39:35.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.200+0000 7fb182f5d640 1 -- 172.21.15.57:0/3914989047 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb17c0b7850 con 0x7fb1600036e0 2024-03-29T10:39:35.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.200+0000 7fb182f5d640 1 -- 172.21.15.57:0/3914989047 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb17c0b5f50 con 0x7fb160007300 2024-03-29T10:39:35.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.200+0000 7fb182f5d640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:35.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.200+0000 7fb182f5d640 10 monclient(hunting): authenticate will time out at 927.376431s 2024-03-29T10:39:35.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.200+0000 7fb1797fa640 10 client.?.objecter ms_handle_connect 0x7fb16000b020 2024-03-29T10:39:35.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.200+0000 7fb1797fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:35.191 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.200+0000 7fb1797fa640 10 client.?.objecter ms_handle_connect 0x7fb160007300 2024-03-29T10:39:35.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.200+0000 7fb1797fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:35.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.200+0000 7fb1797fa640 10 client.?.objecter ms_handle_connect 0x7fb1600036e0 2024-03-29T10:39:35.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.200+0000 7fb1797fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:35.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb1797fa640 1 -- 172.21.15.57:0/3914989047 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102342017 0 0) 0x7fb17c0b7850 con 0x7fb1600036e0 2024-03-29T10:39:35.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb1797fa640 10 monclient(hunting): _negotiate 2024-03-29T10:39:35.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb1797fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:35.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb1797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:35.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb1797fa640 10 monclient(hunting): my global_id is 4108 2024-03-29T10:39:35.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb1797fa640 1 -- 172.21.15.57:0/3914989047 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb160006a20 con 0x7fb1600036e0 2024-03-29T10:39:35.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb1797fa640 1 -- 172.21.15.57:0/3914989047 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3425008924 0 0) 0x7fb160006a20 con 0x7fb1600036e0 2024-03-29T10:39:35.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb1797fa640 1 -- 172.21.15.57:0/3914989047 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb17c0b7850 con 0x7fb1600036e0 2024-03-29T10:39:35.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb1797fa640 1 -- 172.21.15.57:0/3914989047 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb1700018f0 con 0x7fb1600036e0 2024-03-29T10:39:35.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb1797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:35.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb1797fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:35.193 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb1797fa640 10 monclient(hunting): dump: 2024-03-29T10:39:35.193 INFO:teuthology.orchestra.run.smithi057.stderr:epoch 1 2024-03-29T10:39:35.193 INFO:teuthology.orchestra.run.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:35.193 INFO:teuthology.orchestra.run.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:35.193 INFO:teuthology.orchestra.run.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:35.193 INFO:teuthology.orchestra.run.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:35.193 INFO:teuthology.orchestra.run.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:35.193 INFO:teuthology.orchestra.run.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:35.193 INFO:teuthology.orchestra.run.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:35.193 INFO:teuthology.orchestra.run.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:35.193 INFO:teuthology.orchestra.run.smithi057.stderr: 2024-03-29T10:39:35.193 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb1797fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:35.193 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb1797fa640 1 -- 172.21.15.57:0/3914989047 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1913138084 0 0) 0x7fb17c0b7850 con 0x7fb1600036e0 2024-03-29T10:39:35.193 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb1797fa640 1 -- 172.21.15.57:0/3914989047 >> v1:172.21.15.57:6790/0 conn(0x7fb160007300 legacy=0x7fb17c138ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:35.193 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb1797fa640 1 -- 172.21.15.57:0/3914989047 >> v1:172.21.15.57:6789/0 conn(0x7fb16000b020 legacy=0x7fb17c1391f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:35.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb1797fa640 10 monclient: _finish_hunting 0 2024-03-29T10:39:35.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb1797fa640 1 monclient: found mon.b 2024-03-29T10:39:35.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb1797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:35.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb1797fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:35.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb1797fa640 1 -- 172.21.15.57:0/3914989047 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb17c0b60e0 con 0x7fb1600036e0 2024-03-29T10:39:35.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb1797fa640 10 monclient: _finish_auth 0 2024-03-29T10:39:35.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb1797fa640 10 monclient: _check_auth_tickets 2024-03-29T10:39:35.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb1797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:35.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb182f5d640 5 monclient: authenticate success, global_id 4108 2024-03-29T10:39:35.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb182f5d640 10 monclient: _renew_subs 2024-03-29T10:39:35.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb182f5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:35.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.204+0000 7fb182f5d640 1 -- 172.21.15.57:0/3914989047 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb17c0b88b0 con 0x7fb1600036e0 2024-03-29T10:39:35.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.205+0000 7fb182f5d640 10 client.4108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:35.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.205+0000 7fb182f5d640 10 monclient: _renew_subs 2024-03-29T10:39:35.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.205+0000 7fb182f5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:35.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.205+0000 7fb182f5d640 1 -- 172.21.15.57:0/3914989047 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb17c142d10 con 0x7fb1600036e0 2024-03-29T10:39:35.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.205+0000 7fb182f5d640 1 librados: init done 2024-03-29T10:39:35.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.205+0000 7fb1797fa640 1 -- 172.21.15.57:0/3914989047 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1700035f0 con 0x7fb1600036e0 2024-03-29T10:39:35.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.205+0000 7fb1797fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:35.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.205+0000 7fb1797fa640 1 -- 172.21.15.57:0/3914989047 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb170003900 con 0x7fb1600036e0 2024-03-29T10:39:35.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.205+0000 7fb1797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:35.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.205+0000 7fb1797fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:35.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.205+0000 7fb17affd640 4 set_mon_vals no callback set 2024-03-29T10:39:35.195 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.205+0000 7fb1797fa640 10 monclient: dump: 2024-03-29T10:39:35.195 INFO:teuthology.orchestra.run.smithi057.stderr:epoch 1 2024-03-29T10:39:35.195 INFO:teuthology.orchestra.run.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:35.195 INFO:teuthology.orchestra.run.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:35.195 INFO:teuthology.orchestra.run.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:35.195 INFO:teuthology.orchestra.run.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:35.196 INFO:teuthology.orchestra.run.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:35.196 INFO:teuthology.orchestra.run.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:35.196 INFO:teuthology.orchestra.run.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:35.196 INFO:teuthology.orchestra.run.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:35.196 INFO:teuthology.orchestra.run.smithi057.stderr: 2024-03-29T10:39:35.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.205+0000 7fb1797fa640 1 -- 172.21.15.57:0/3914989047 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb170003df0 con 0x7fb1600036e0 2024-03-29T10:39:35.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.205+0000 7fb1797fa640 1 -- 172.21.15.57:0/3914989047 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2929380104 0 0) 0x7fb170005330 con 0x7fb1600036e0 2024-03-29T10:39:35.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.205+0000 7fb1797fa640 10 client.4108.objecter ms_dispatch 0x7fb17c000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T10:39:35.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.205+0000 7fb1797fa640 3 client.4108.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T10:39:35.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.205+0000 7fb1797fa640 3 client.4108.objecter handle_osd_map decoding full epoch 1 2024-03-29T10:39:35.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.205+0000 7fb1797fa640 20 client.4108.objecter dump_active .. 0 homeless 2024-03-29T10:39:35.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.206+0000 7fb182f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:35.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.206+0000 7fb182f5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:35.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.206+0000 7fb182f5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:35.196 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.206+0000 7fb182f5d640 1 -- 172.21.15.57:0/3914989047 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb15c005080 con 0x7fb1600036e0 2024-03-29T10:39:35.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.208+0000 7fb1797fa640 1 -- 172.21.15.57:0/3914989047 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb170004230 con 0x7fb1600036e0 2024-03-29T10:39:35.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.208+0000 7fb1797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:35.197 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.208+0000 7fb1797fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:35.263 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.280+0000 7fb182f5d640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:39:35.264 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.280+0000 7fb182f5d640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:39:35.264 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.280+0000 7fb182f5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:35.264 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.280+0000 7fb182f5d640 1 -- 172.21.15.57:0/3914989047 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fb15c005330 con 0x7fb1600036e0 2024-03-29T10:39:35.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.288+0000 7fb1797fa640 1 -- 172.21.15.57:0/3914989047 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fb170004530 con 0x7fb1600036e0 2024-03-29T10:39:35.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.288+0000 7fb1797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:39:35.271 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.288+0000 7fb1797fa640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T10:39:35.271 INFO:teuthology.orchestra.run.smithi057.stderr:adjusted tunables profile to default 2024-03-29T10:39:35.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.288+0000 7fb182f5d640 10 librados: watch_flush enter 2024-03-29T10:39:35.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.289+0000 7fb182f5d640 10 librados: watch_flush exit 2024-03-29T10:39:35.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.289+0000 7fb182f5d640 20 client.4108.objecter shutdown clearing up homeless session... 2024-03-29T10:39:35.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.289+0000 7fb182f5d640 10 client.4108.objecter successfully canceled tick 2024-03-29T10:39:35.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.289+0000 7fb182f5d640 10 monclient: shutdown 2024-03-29T10:39:35.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.289+0000 7fb182f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:35.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.289+0000 7fb182f5d640 1 -- 172.21.15.57:0/3914989047 >> v1:172.21.15.179:6789/0 conn(0x7fb1600036e0 legacy=0x7fb17c0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:35.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.289+0000 7fb182f5d640 1 -- 172.21.15.57:0/3914989047 shutdown_connections 2024-03-29T10:39:35.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.289+0000 7fb182f5d640 1 -- 172.21.15.57:0/3914989047 >> 172.21.15.57:0/3914989047 conn(0x7fb17c066b00 msgr2=0x7fb17c13c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:35.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.289+0000 7fb182f5d640 1 -- 172.21.15.57:0/3914989047 shutdown_connections 2024-03-29T10:39:35.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.289+0000 7fb182f5d640 1 -- 172.21.15.57:0/3914989047 wait complete. 2024-03-29T10:39:35.273 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:35.289+0000 7fb182f5d640 1 librados: shutdown 2024-03-29T10:39:35.283 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T10:39:35.283 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T10:39:35.284 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-29T10:39:35.284 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T10:39:35.312 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-29T10:39:35.312 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T10:39:35.379 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-29T10:39:35.379 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T10:39:35.446 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-29T10:39:35.446 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T10:39:35.513 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T10:39:35.513 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T10:39:35.540 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T10:39:35.541 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T10:39:35.608 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T10:39:35.608 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T10:39:35.665 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:35.681+0000 7f4aa0845040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:39:35.677 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-29T10:39:35.678 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T10:39:35.714 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:35.731+0000 7f8972a32040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:39:35.721 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:35.737+0000 7f4aa0845040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:39:35.745 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4 0 2024-03-29T10:39:35.770 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:35.786+0000 7f8972a32040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:39:35.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.901+0000 7fe2bb92b640 10 monclient: get_monmap_and_config 2024-03-29T10:39:35.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.901+0000 7fe2bb92b640 10 monclient: build_initial_monmap 2024-03-29T10:39:35.885 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.901+0000 7fe2bb92b640 10 monclient: monmap: 2024-03-29T10:39:35.885 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T10:39:35.885 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:35.885 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:39:35.902160+0000 2024-03-29T10:39:35.885 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:39:35.902160+0000 2024-03-29T10:39:35.885 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:35.885 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:35.885 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:35.885 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:39:35.886 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:39:35.886 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:35.895 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.901+0000 7fe2bb92b640 1 Processor -- start 2024-03-29T10:39:35.895 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.901+0000 7fe2bb92b640 1 -- start start 2024-03-29T10:39:35.896 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.901+0000 7fe2bb92b640 10 monclient: init 2024-03-29T10:39:35.896 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.902+0000 7fe2bb92b640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:35.896 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.902+0000 7fe2bb92b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:39:35.896 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.902+0000 7fe2bb92b640 10 monclient(hunting): picked mon.noname-b con 0x7fe2b409ac40 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:35.896 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.902+0000 7fe2bb92b640 10 monclient(hunting): picked mon.noname-a con 0x7fe2b409c870 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:35.896 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.902+0000 7fe2bb92b640 10 monclient(hunting): picked mon.noname-c con 0x7fe2b40b4360 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:35.896 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.902+0000 7fe2bb92b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b409dec0 con 0x7fe2b409c870 2024-03-29T10:39:35.896 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.902+0000 7fe2bb92b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b409da30 con 0x7fe2b40b4360 2024-03-29T10:39:35.896 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.902+0000 7fe2bb92b640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b40b39c0 con 0x7fe2b409ac40 2024-03-29T10:39:35.896 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.902+0000 7fe2bb92b640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:35.896 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.902+0000 7fe2bb92b640 10 monclient(hunting): authenticate will time out at 1088.171658s 2024-03-29T10:39:35.896 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.902+0000 7fe2b9ec8640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fe2b40b4360 0x7fe2b414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:35094/0 (socket says 172.21.15.179:35094) 2024-03-29T10:39:35.896 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.902+0000 7fe2b9ec8640 1 -- 172.21.15.179:0/2163608582 learned_addr learned my addr 172.21.15.179:0/2163608582 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T10:39:35.897 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.902+0000 7fe2abfff640 1 -- 172.21.15.179:0/2163608582 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741409645 0 0) 0x7fe2b409da30 con 0x7fe2b40b4360 2024-03-29T10:39:35.897 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.902+0000 7fe2abfff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:35.897 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.902+0000 7fe2abfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:35.897 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.902+0000 7fe2abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:35.897 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.902+0000 7fe2abfff640 10 monclient(hunting): my global_id is 4111 2024-03-29T10:39:35.897 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.902+0000 7fe2abfff640 1 -- 172.21.15.179:0/2163608582 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2a0003520 con 0x7fe2b40b4360 2024-03-29T10:39:35.897 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 1 -- 172.21.15.179:0/2163608582 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915121902 0 0) 0x7fe2a4001110 con 0x7fe2b409c870 2024-03-29T10:39:35.897 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:35.897 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:35.897 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:35.897 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 10 monclient(hunting): my global_id is 4110 2024-03-29T10:39:35.897 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 1 -- 172.21.15.179:0/2163608582 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2b409da30 con 0x7fe2b409c870 2024-03-29T10:39:35.897 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 1 -- 172.21.15.179:0/2163608582 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3832066777 0 0) 0x7fe2b40b39c0 con 0x7fe2b409ac40 2024-03-29T10:39:35.897 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:35.897 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:35.897 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:35.898 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 10 monclient(hunting): my global_id is 4115 2024-03-29T10:39:35.898 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 1 -- 172.21.15.179:0/2163608582 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2a4001110 con 0x7fe2b409ac40 2024-03-29T10:39:35.898 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 1 -- 172.21.15.179:0/2163608582 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1152342842 0 0) 0x7fe2a0003520 con 0x7fe2b40b4360 2024-03-29T10:39:35.898 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 1 -- 172.21.15.179:0/2163608582 >> v1:172.21.15.57:6790/0 conn(0x7fe2b409ac40 legacy=0x7fe2b40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:35.898 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 1 -- 172.21.15.179:0/2163608582 >> v1:172.21.15.57:6789/0 conn(0x7fe2b409c870 legacy=0x7fe2b413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:35.898 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 10 monclient: _finish_hunting 0 2024-03-29T10:39:35.898 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 1 monclient: found mon.noname-c 2024-03-29T10:39:35.898 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:35.898 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:39:35.898 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 1 -- 172.21.15.179:0/2163608582 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b40b5160 con 0x7fe2b40b4360 2024-03-29T10:39:35.898 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 10 monclient: _finish_auth 0 2024-03-29T10:39:35.898 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 10 monclient: _check_auth_tickets 2024-03-29T10:39:35.898 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:35.898 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 1 -- 172.21.15.179:0/2163608582 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fe2ac001a10 con 0x7fe2b40b4360 2024-03-29T10:39:35.899 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2bb92b640 5 monclient: authenticate success, global_id 4111 2024-03-29T10:39:35.899 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2bb92b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:35.899 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:35.899 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:35.899 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 10 monclient: dump: 2024-03-29T10:39:35.899 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:35.899 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:35.899 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:35.899 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:35.899 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:35.899 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:35.899 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:35.899 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:35.899 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:35.899 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:35.899 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2bb92b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:35.900 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 1 -- 172.21.15.179:0/2163608582 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2ac001bf0 con 0x7fe2b40b4360 2024-03-29T10:39:35.900 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:35.900 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 1 -- 172.21.15.179:0/2163608582 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fe2ac002000 con 0x7fe2b40b4360 2024-03-29T10:39:35.900 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2bb92b640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:35.900 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2bb92b640 4 set_mon_vals no callback set 2024-03-29T10:39:35.900 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:35.900 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:35.900 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2abfff640 10 monclient: dump: 2024-03-29T10:39:35.900 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:35.900 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:35.900 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:35.900 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:35.900 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:35.900 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:35.900 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:35.900 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:35.901 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:35.901 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:35.901 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2bb92b640 10 monclient: shutdown 2024-03-29T10:39:35.901 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2bb92b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:35.901 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2bb92b640 1 -- 172.21.15.179:0/2163608582 >> v1:172.21.15.179:6789/0 conn(0x7fe2b40b4360 legacy=0x7fe2b414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:35.901 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2bb92b640 1 -- 172.21.15.179:0/2163608582 shutdown_connections 2024-03-29T10:39:35.901 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.903+0000 7fe2bb92b640 1 -- 172.21.15.179:0/2163608582 >> 172.21.15.179:0/2163608582 conn(0x7fe2b4066b00 msgr2=0x7fe2b4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:35.901 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.904+0000 7fe2bb92b640 1 -- 172.21.15.179:0/2163608582 shutdown_connections 2024-03-29T10:39:35.901 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.904+0000 7fe2bb92b640 1 -- 172.21.15.179:0/2163608582 wait complete. 2024-03-29T10:39:35.901 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.904+0000 7fe2bb92b640 10 monclient: build_initial_monmap 2024-03-29T10:39:35.901 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.904+0000 7fe2bb92b640 10 monclient: monmap: 2024-03-29T10:39:35.901 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T10:39:35.901 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:35.901 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 0.000000 2024-03-29T10:39:35.901 INFO:teuthology.orchestra.run.smithi179.stderr:created 0.000000 2024-03-29T10:39:35.902 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:35.902 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:35.902 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:35.902 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:39:35.902 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:39:35.902 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:35.902 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.904+0000 7fe2bb92b640 1 librados: starting msgr at 2024-03-29T10:39:35.902 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.904+0000 7fe2bb92b640 1 librados: starting objecter 2024-03-29T10:39:35.902 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.904+0000 7fe2bb92b640 1 Processor -- start 2024-03-29T10:39:35.902 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2bb92b640 1 -- start start 2024-03-29T10:39:35.902 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2bb92b640 1 librados: setting wanted keys 2024-03-29T10:39:35.902 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2bb92b640 1 librados: calling monclient init 2024-03-29T10:39:35.902 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2bb92b640 10 monclient: init 2024-03-29T10:39:35.902 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2bb92b640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:35.902 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2bb92b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:39:35.903 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2bb92b640 10 monclient(hunting): picked mon.noname-a con 0x7fe2b409ac40 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:35.903 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2bb92b640 10 monclient(hunting): picked mon.noname-b con 0x7fe2b4140a10 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:35.903 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2bb92b640 10 monclient(hunting): picked mon.noname-c con 0x7fe2b41d7e70 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:35.903 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2bb92b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b41d7280 con 0x7fe2b409ac40 2024-03-29T10:39:35.903 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2bb92b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b4142b20 con 0x7fe2b4140a10 2024-03-29T10:39:35.903 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2bb92b640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b4144420 con 0x7fe2b41d7e70 2024-03-29T10:39:35.904 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2bb92b640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:35.904 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2bb92b640 10 monclient(hunting): authenticate will time out at 1088.174654s 2024-03-29T10:39:35.904 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2b8ec6640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fe2b4140a10 0x7fe2b41d7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:35100/0 (socket says 172.21.15.179:35100) 2024-03-29T10:39:35.904 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2b8ec6640 1 -- 172.21.15.179:0/1601439604 learned_addr learned my addr 172.21.15.179:0/1601439604 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T10:39:35.904 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2a9ffb640 10 client.?.objecter ms_handle_connect 0x7fe2b4140a10 2024-03-29T10:39:35.905 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:35.905 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2a9ffb640 10 client.?.objecter ms_handle_connect 0x7fe2b409ac40 2024-03-29T10:39:35.905 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:35.905 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2a9ffb640 10 client.?.objecter ms_handle_connect 0x7fe2b41d7e70 2024-03-29T10:39:35.905 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:35.905 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102059313 0 0) 0x7fe2b4142b20 con 0x7fe2b4140a10 2024-03-29T10:39:35.905 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:35.905 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:35.905 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:35.905 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2a9ffb640 10 monclient(hunting): my global_id is 4114 2024-03-29T10:39:35.905 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe298004eb0 con 0x7fe2b4140a10 2024-03-29T10:39:35.906 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 708564255 0 0) 0x7fe2b41d7280 con 0x7fe2b409ac40 2024-03-29T10:39:35.906 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:35.906 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:35.906 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:35.906 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2a9ffb640 10 monclient(hunting): my global_id is 4113 2024-03-29T10:39:35.906 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2b4142b20 con 0x7fe2b409ac40 2024-03-29T10:39:35.906 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732576849 0 0) 0x7fe2b4144420 con 0x7fe2b41d7e70 2024-03-29T10:39:35.906 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:35.906 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:35.906 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:35.906 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2a9ffb640 10 monclient(hunting): my global_id is 4118 2024-03-29T10:39:35.906 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.905+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2b41d7280 con 0x7fe2b41d7e70 2024-03-29T10:39:35.906 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 886340924 0 0) 0x7fe298004eb0 con 0x7fe2b4140a10 2024-03-29T10:39:35.907 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2b4144420 con 0x7fe2b4140a10 2024-03-29T10:39:35.907 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fe2a40019a0 con 0x7fe2b4140a10 2024-03-29T10:39:35.907 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:35.907 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:35.907 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 10 monclient(hunting): dump: 2024-03-29T10:39:35.907 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:35.907 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:35.907 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:35.907 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:35.907 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:35.907 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:35.907 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:35.907 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:35.908 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:35.908 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:35.908 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:35.908 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 322658221 0 0) 0x7fe2b4142b20 con 0x7fe2b409ac40 2024-03-29T10:39:35.908 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe298004eb0 con 0x7fe2b409ac40 2024-03-29T10:39:35.908 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fe2b00017e0 con 0x7fe2b409ac40 2024-03-29T10:39:35.908 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:35.908 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:35.908 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 10 monclient(hunting): dump: 2024-03-29T10:39:35.908 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:35.908 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:35.908 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:35.908 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:35.908 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:35.909 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:35.909 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:35.909 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:35.909 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:35.909 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:35.909 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3842038975 0 0) 0x7fe2b41d7280 con 0x7fe2b41d7e70 2024-03-29T10:39:35.909 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2b4142b20 con 0x7fe2b41d7e70 2024-03-29T10:39:35.909 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fe2ac001f00 con 0x7fe2b41d7e70 2024-03-29T10:39:35.909 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:35.909 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:35.909 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 10 monclient(hunting): dump: 2024-03-29T10:39:35.909 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:35.909 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:35.909 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:35.909 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:35.910 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:35.910 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:35.910 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:35.910 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:35.910 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:35.910 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:35.910 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3936308333 0 0) 0x7fe2b4144420 con 0x7fe2b4140a10 2024-03-29T10:39:35.910 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 >> v1:172.21.15.57:6790/0 conn(0x7fe2b41d7e70 legacy=0x7fe2b40b0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:35.910 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 >> v1:172.21.15.57:6789/0 conn(0x7fe2b409ac40 legacy=0x7fe2b4141690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:35.910 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:39:35.910 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 1 monclient: found mon.b 2024-03-29T10:39:35.910 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:35.910 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:35.910 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b41445b0 con 0x7fe2b4140a10 2024-03-29T10:39:35.911 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 10 monclient: _finish_auth 0 2024-03-29T10:39:35.911 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:39:35.911 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:35.911 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2bb92b640 5 monclient: authenticate success, global_id 4114 2024-03-29T10:39:35.911 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2bb92b640 10 monclient: _renew_subs 2024-03-29T10:39:35.911 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2bb92b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:35.911 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2bb92b640 1 -- 172.21.15.179:0/1601439604 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b4140e70 con 0x7fe2b4140a10 2024-03-29T10:39:35.911 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2a4001c40 con 0x7fe2b4140a10 2024-03-29T10:39:35.911 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:35.911 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fe2a40038c0 con 0x7fe2b4140a10 2024-03-29T10:39:35.911 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:35.911 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:35.911 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 10 monclient: dump: 2024-03-29T10:39:35.911 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:35.911 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:35.912 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:35.912 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:35.912 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:35.912 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:35.912 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:35.912 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:35.912 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:35.912 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:35.912 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe2a4003e30 con 0x7fe2b4140a10 2024-03-29T10:39:35.912 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2bb92b640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:35.912 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2ab7fe640 4 set_mon_vals no callback set 2024-03-29T10:39:35.912 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2bb92b640 10 monclient: _renew_subs 2024-03-29T10:39:35.912 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2bb92b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:35.912 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2bb92b640 1 -- 172.21.15.179:0/1601439604 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b40b1860 con 0x7fe2b4140a10 2024-03-29T10:39:35.912 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2bb92b640 1 librados: init done 2024-03-29T10:39:35.913 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 3476982942 0 0) 0x7fe2a4005b00 con 0x7fe2b4140a10 2024-03-29T10:39:35.913 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 10 client.4114.objecter ms_dispatch 0x7fe2b4000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T10:39:35.913 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 3 client.4114.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T10:39:35.913 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 3 client.4114.objecter handle_osd_map decoding full epoch 2 2024-03-29T10:39:35.913 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.906+0000 7fe2a9ffb640 20 client.4114.objecter dump_active .. 0 homeless 2024-03-29T10:39:35.913 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.907+0000 7fe2bb92b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:35.913 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.907+0000 7fe2bb92b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:35.913 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.907+0000 7fe2bb92b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:35.913 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.907+0000 7fe2bb92b640 1 -- 172.21.15.179:0/1601439604 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b41d7480 con 0x7fe2b4140a10 2024-03-29T10:39:35.913 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.909+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe2b41d7480 con 0x7fe2b4140a10 2024-03-29T10:39:35.913 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.909+0000 7fe2a9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:35.913 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.909+0000 7fe2a9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:35.964 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.980+0000 7fe2bb92b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4", "id": 0}] 2024-03-29T10:39:35.964 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.980+0000 7fe2bb92b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4", "id": 0}] 2024-03-29T10:39:35.964 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.980+0000 7fe2bb92b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:35.964 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.980+0000 7fe2bb92b640 1 -- 172.21.15.179:0/1601439604 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4", "id": 0} v 0) v1 -- 0x7fe2b4143220 con 0x7fe2b4140a10 2024-03-29T10:39:35.978 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.994+0000 7fe2a9ffb640 1 -- 172.21.15.179:0/1601439604 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 2687274265 0 732707515) 0x7fe2a4004310 con 0x7fe2b4140a10 2024-03-29T10:39:35.978 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.994+0000 7fe2a9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4", "id": 0}] 2024-03-29T10:39:35.979 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.994+0000 7fe2a9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:35.979 INFO:teuthology.orchestra.run.smithi179.stdout:0 2024-03-29T10:39:35.980 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.995+0000 7fe2bb92b640 10 librados: watch_flush enter 2024-03-29T10:39:35.980 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.995+0000 7fe2bb92b640 10 librados: watch_flush exit 2024-03-29T10:39:35.980 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.995+0000 7fe2bb92b640 20 client.4114.objecter shutdown clearing up homeless session... 2024-03-29T10:39:35.980 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.995+0000 7fe2bb92b640 10 client.4114.objecter successfully canceled tick 2024-03-29T10:39:35.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.995+0000 7fe2bb92b640 10 monclient: shutdown 2024-03-29T10:39:35.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.995+0000 7fe2bb92b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:35.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.995+0000 7fe2bb92b640 1 -- 172.21.15.179:0/1601439604 >> v1:172.21.15.179:6789/0 conn(0x7fe2b4140a10 legacy=0x7fe2b41d7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:35.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.995+0000 7fe2bb92b640 1 -- 172.21.15.179:0/1601439604 shutdown_connections 2024-03-29T10:39:35.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.995+0000 7fe2bb92b640 1 -- 172.21.15.179:0/1601439604 >> 172.21.15.179:0/1601439604 conn(0x7fe2b4066b00 msgr2=0x7fe2b413d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:35.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.995+0000 7fe2bb92b640 1 -- 172.21.15.179:0/1601439604 shutdown_connections 2024-03-29T10:39:35.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.995+0000 7fe2bb92b640 1 -- 172.21.15.179:0/1601439604 wait complete. 2024-03-29T10:39:35.981 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:35.995+0000 7fe2bb92b640 1 librados: shutdown 2024-03-29T10:39:35.990 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new e1569630-d46c-4cc7-93c7-73c7ad2140ac 1 2024-03-29T10:39:36.090 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.106+0000 7f68da35d640 10 monclient: get_monmap_and_config 2024-03-29T10:39:36.090 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.106+0000 7f68da35d640 10 monclient: build_initial_monmap 2024-03-29T10:39:36.090 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.106+0000 7f68da35d640 10 monclient: monmap: 2024-03-29T10:39:36.090 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T10:39:36.090 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.090 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:39:36.107244+0000 2024-03-29T10:39:36.091 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:39:36.107244+0000 2024-03-29T10:39:36.091 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:36.091 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.091 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:36.091 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:39:36.091 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:39:36.091 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.101 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.107+0000 7f68da35d640 1 Processor -- start 2024-03-29T10:39:36.101 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.107+0000 7f68da35d640 1 -- start start 2024-03-29T10:39:36.101 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.107+0000 7f68da35d640 10 monclient: init 2024-03-29T10:39:36.101 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.107+0000 7f68da35d640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:36.101 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.107+0000 7f68da35d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:39:36.101 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.107+0000 7f68da35d640 10 monclient(hunting): picked mon.noname-c con 0x7f68d40af360 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:36.101 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.107+0000 7f68da35d640 10 monclient(hunting): picked mon.noname-a con 0x7f68d406af60 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:36.101 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.107+0000 7f68da35d640 10 monclient(hunting): picked mon.noname-b con 0x7f68d4145e60 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:36.101 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.107+0000 7f68da35d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d40b0520 con 0x7f68d406af60 2024-03-29T10:39:36.101 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.107+0000 7f68da35d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d40aff20 con 0x7f68d40af360 2024-03-29T10:39:36.101 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.107+0000 7f68da35d640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d40b1da0 con 0x7f68d4145e60 2024-03-29T10:39:36.101 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.107+0000 7f68da35d640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:36.102 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.107+0000 7f68da35d640 10 monclient(hunting): authenticate will time out at 1088.376747s 2024-03-29T10:39:36.102 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.107+0000 7f68d3fff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f68d40af360 0x7f68d406c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:35112/0 (socket says 172.21.15.179:35112) 2024-03-29T10:39:36.102 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.107+0000 7f68d3fff640 1 -- 172.21.15.179:0/952332034 learned_addr learned my addr 172.21.15.179:0/952332034 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T10:39:36.102 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.107+0000 7f68d2ffd640 1 -- 172.21.15.179:0/952332034 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157789602 0 0) 0x7f68d40aff20 con 0x7f68d40af360 2024-03-29T10:39:36.102 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.107+0000 7f68d2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.102 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.107+0000 7f68d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.102 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.107+0000 7f68d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.102 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.107+0000 7f68d2ffd640 10 monclient(hunting): my global_id is 4117 2024-03-29T10:39:36.102 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 1 -- 172.21.15.179:0/952332034 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68bc003520 con 0x7f68d40af360 2024-03-29T10:39:36.102 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 1 -- 172.21.15.179:0/952332034 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2583327298 0 0) 0x7f68d40b1da0 con 0x7f68d4145e60 2024-03-29T10:39:36.102 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.102 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.102 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.102 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 10 monclient(hunting): my global_id is 4121 2024-03-29T10:39:36.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 1 -- 172.21.15.179:0/952332034 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68d40aff20 con 0x7f68d4145e60 2024-03-29T10:39:36.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 1 -- 172.21.15.179:0/952332034 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3254943697 0 0) 0x7f68d40b0520 con 0x7f68d406af60 2024-03-29T10:39:36.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 10 monclient(hunting): my global_id is 4116 2024-03-29T10:39:36.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 1 -- 172.21.15.179:0/952332034 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68d40b1da0 con 0x7f68d406af60 2024-03-29T10:39:36.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 1 -- 172.21.15.179:0/952332034 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1984983775 0 0) 0x7f68bc003520 con 0x7f68d40af360 2024-03-29T10:39:36.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 1 -- 172.21.15.179:0/952332034 >> v1:172.21.15.57:6790/0 conn(0x7f68d4145e60 legacy=0x7f68d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 1 -- 172.21.15.179:0/952332034 >> v1:172.21.15.57:6789/0 conn(0x7f68d406af60 legacy=0x7f68d4145750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:39:36.103 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 1 monclient: found mon.noname-c 2024-03-29T10:39:36.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:36.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 1 -- 172.21.15.179:0/952332034 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d40b2080 con 0x7f68d40af360 2024-03-29T10:39:36.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:39:36.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:39:36.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:36.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 1 -- 172.21.15.179:0/952332034 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f68c4001690 con 0x7f68d40af360 2024-03-29T10:39:36.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68da35d640 5 monclient: authenticate success, global_id 4117 2024-03-29T10:39:36.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68da35d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:36.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:36.104 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 10 monclient: dump: 2024-03-29T10:39:36.104 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.104 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.104 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.105 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.105 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.105 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.105 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.105 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.105 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.105 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68da35d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:36.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 1 -- 172.21.15.179:0/952332034 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68c40018b0 con 0x7f68d40af360 2024-03-29T10:39:36.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:36.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68da35d640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:36.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 1 -- 172.21.15.179:0/952332034 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f68c4001bc0 con 0x7f68d40af360 2024-03-29T10:39:36.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68da35d640 4 set_mon_vals no callback set 2024-03-29T10:39:36.106 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:36.106 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68d2ffd640 10 monclient: dump: 2024-03-29T10:39:36.106 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.106 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.106 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.106 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.106 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.106 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.106 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.106 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.106 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.107 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.107 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68da35d640 10 monclient: shutdown 2024-03-29T10:39:36.107 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68da35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:36.107 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68da35d640 1 -- 172.21.15.179:0/952332034 >> v1:172.21.15.179:6789/0 conn(0x7f68d40af360 legacy=0x7f68d406c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.107 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68da35d640 1 -- 172.21.15.179:0/952332034 shutdown_connections 2024-03-29T10:39:36.107 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.108+0000 7f68da35d640 1 -- 172.21.15.179:0/952332034 >> 172.21.15.179:0/952332034 conn(0x7f68d40b58f0 msgr2=0x7f68d40b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:36.107 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.109+0000 7f68da35d640 1 -- 172.21.15.179:0/952332034 shutdown_connections 2024-03-29T10:39:36.107 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.109+0000 7f68da35d640 1 -- 172.21.15.179:0/952332034 wait complete. 2024-03-29T10:39:36.107 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.109+0000 7f68da35d640 10 monclient: build_initial_monmap 2024-03-29T10:39:36.107 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.109+0000 7f68da35d640 10 monclient: monmap: 2024-03-29T10:39:36.107 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T10:39:36.108 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:36.108 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 0.000000 2024-03-29T10:39:36.108 INFO:teuthology.orchestra.run.smithi179.stderr:created 0.000000 2024-03-29T10:39:36.108 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:36.108 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.108 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:36.108 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:39:36.108 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:39:36.108 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.108 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.109+0000 7f68da35d640 1 librados: starting msgr at 2024-03-29T10:39:36.108 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.109+0000 7f68da35d640 1 librados: starting objecter 2024-03-29T10:39:36.108 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.109+0000 7f68da35d640 1 Processor -- start 2024-03-29T10:39:36.108 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68da35d640 1 -- start start 2024-03-29T10:39:36.108 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68da35d640 1 librados: setting wanted keys 2024-03-29T10:39:36.108 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68da35d640 1 librados: calling monclient init 2024-03-29T10:39:36.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68da35d640 10 monclient: init 2024-03-29T10:39:36.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68da35d640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:36.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68da35d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:39:36.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68da35d640 10 monclient(hunting): picked mon.noname-b con 0x7f68d406af60 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:36.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68da35d640 10 monclient(hunting): picked mon.noname-a con 0x7f68d40af360 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:36.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68da35d640 10 monclient(hunting): picked mon.noname-c con 0x7f68d4145e60 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:36.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68da35d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d41216a0 con 0x7f68d40af360 2024-03-29T10:39:36.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68da35d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d411ff10 con 0x7f68d406af60 2024-03-29T10:39:36.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68da35d640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d411fb30 con 0x7f68d4145e60 2024-03-29T10:39:36.109 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68da35d640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:36.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68da35d640 10 monclient(hunting): authenticate will time out at 1088.379733s 2024-03-29T10:39:36.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68d3fff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f68d406af60 0x7f68d41d6e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:35116/0 (socket says 172.21.15.179:35116) 2024-03-29T10:39:36.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68d3fff640 1 -- 172.21.15.179:0/1833069303 learned_addr learned my addr 172.21.15.179:0/1833069303 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T10:39:36.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68d0ff9640 10 client.?.objecter ms_handle_connect 0x7f68d406af60 2024-03-29T10:39:36.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:36.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68d0ff9640 10 client.?.objecter ms_handle_connect 0x7f68d40af360 2024-03-29T10:39:36.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:36.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68d0ff9640 1 -- 172.21.15.179:0/1833069303 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215001510 0 0) 0x7f68d411ff10 con 0x7f68d406af60 2024-03-29T10:39:36.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68d0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68d0ff9640 10 monclient(hunting): my global_id is 4120 2024-03-29T10:39:36.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68d0ff9640 1 -- 172.21.15.179:0/1833069303 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68c0004eb0 con 0x7f68d406af60 2024-03-29T10:39:36.110 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68d0ff9640 10 client.?.objecter ms_handle_connect 0x7f68d4145e60 2024-03-29T10:39:36.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.110+0000 7f68d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:36.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 1 -- 172.21.15.179:0/1833069303 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085012306 0 0) 0x7f68d41216a0 con 0x7f68d40af360 2024-03-29T10:39:36.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 10 monclient(hunting): my global_id is 4119 2024-03-29T10:39:36.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 1 -- 172.21.15.179:0/1833069303 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68d411ff10 con 0x7f68d40af360 2024-03-29T10:39:36.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 1 -- 172.21.15.179:0/1833069303 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774618341 0 0) 0x7f68d411fb30 con 0x7f68d4145e60 2024-03-29T10:39:36.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.111 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 10 monclient(hunting): my global_id is 4124 2024-03-29T10:39:36.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 1 -- 172.21.15.179:0/1833069303 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68d41216a0 con 0x7f68d4145e60 2024-03-29T10:39:36.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 1 -- 172.21.15.179:0/1833069303 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3795120333 0 0) 0x7f68c0004eb0 con 0x7f68d406af60 2024-03-29T10:39:36.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 1 -- 172.21.15.179:0/1833069303 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68d411fb30 con 0x7f68d406af60 2024-03-29T10:39:36.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 1 -- 172.21.15.179:0/1833069303 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f68c4001610 con 0x7f68d406af60 2024-03-29T10:39:36.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:36.112 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 10 monclient(hunting): dump: 2024-03-29T10:39:36.112 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.112 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.112 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.112 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.112 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.112 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.112 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.113 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.113 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.113 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:36.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 1 -- 172.21.15.179:0/1833069303 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3415954665 0 0) 0x7f68d411ff10 con 0x7f68d40af360 2024-03-29T10:39:36.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 1 -- 172.21.15.179:0/1833069303 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68c0004eb0 con 0x7f68d40af360 2024-03-29T10:39:36.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 1 -- 172.21.15.179:0/1833069303 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3614112497 0 0) 0x7f68d41216a0 con 0x7f68d4145e60 2024-03-29T10:39:36.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 1 -- 172.21.15.179:0/1833069303 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68d411ff10 con 0x7f68d4145e60 2024-03-29T10:39:36.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 1 -- 172.21.15.179:0/1833069303 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1251597854 0 0) 0x7f68d411fb30 con 0x7f68d406af60 2024-03-29T10:39:36.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 1 -- 172.21.15.179:0/1833069303 >> v1:172.21.15.57:6790/0 conn(0x7f68d4145e60 legacy=0x7f68d41e2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 1 -- 172.21.15.179:0/1833069303 >> v1:172.21.15.57:6789/0 conn(0x7f68d40af360 legacy=0x7f68d413e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:39:36.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 1 monclient: found mon.b 2024-03-29T10:39:36.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:36.113 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.114 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 1 -- 172.21.15.179:0/1833069303 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d411fcc0 con 0x7f68d406af60 2024-03-29T10:39:36.114 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 10 monclient: _finish_auth 0 2024-03-29T10:39:36.114 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:39:36.114 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:36.114 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68da35d640 5 monclient: authenticate success, global_id 4120 2024-03-29T10:39:36.114 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68da35d640 10 monclient: _renew_subs 2024-03-29T10:39:36.114 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68da35d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.114 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68da35d640 1 -- 172.21.15.179:0/1833069303 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d4120430 con 0x7f68d406af60 2024-03-29T10:39:36.114 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68da35d640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:36.114 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68da35d640 10 monclient: _renew_subs 2024-03-29T10:39:36.114 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68da35d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.115 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68da35d640 1 -- 172.21.15.179:0/1833069303 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d41d7700 con 0x7f68d406af60 2024-03-29T10:39:36.115 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68da35d640 1 librados: init done 2024-03-29T10:39:36.115 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 1 -- 172.21.15.179:0/1833069303 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68c4004a90 con 0x7f68d406af60 2024-03-29T10:39:36.115 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:36.115 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 1 -- 172.21.15.179:0/1833069303 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f68c4001240 con 0x7f68d406af60 2024-03-29T10:39:36.115 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.115 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:36.115 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 10 monclient: dump: 2024-03-29T10:39:36.115 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.115 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.115 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.115 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.115 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.115 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.115 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.116 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.116 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.116 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d27fc640 4 set_mon_vals no callback set 2024-03-29T10:39:36.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 1 -- 172.21.15.179:0/1833069303 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f68c4005130 con 0x7f68d406af60 2024-03-29T10:39:36.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 1 -- 172.21.15.179:0/1833069303 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 280636105 0 0) 0x7f68c4006710 con 0x7f68d406af60 2024-03-29T10:39:36.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 10 client.4120.objecter ms_dispatch 0x7f68d4000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T10:39:36.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 3 client.4120.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T10:39:36.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 3 client.4120.objecter handle_osd_map decoding full epoch 3 2024-03-29T10:39:36.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.111+0000 7f68d0ff9640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-29T10:39:36.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.112+0000 7f68da35d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:36.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.112+0000 7f68da35d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:36.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.112+0000 7f68da35d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.112+0000 7f68da35d640 1 -- 172.21.15.179:0/1833069303 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b4005080 con 0x7f68d406af60 2024-03-29T10:39:36.116 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.114+0000 7f68d0ff9640 1 -- 172.21.15.179:0/1833069303 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f68c40069f0 con 0x7f68d406af60 2024-03-29T10:39:36.117 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.114+0000 7f68d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:36.117 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.114+0000 7f68d0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:36.170 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.186+0000 7f68da35d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e1569630-d46c-4cc7-93c7-73c7ad2140ac", "id": 1}] 2024-03-29T10:39:36.170 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.186+0000 7f68da35d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e1569630-d46c-4cc7-93c7-73c7ad2140ac", "id": 1}] 2024-03-29T10:39:36.170 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.186+0000 7f68da35d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.170 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.186+0000 7f68da35d640 1 -- 172.21.15.179:0/1833069303 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd new", "uuid": "e1569630-d46c-4cc7-93c7-73c7ad2140ac", "id": 1} v 0) v1 -- 0x7f68b40058b0 con 0x7f68d406af60 2024-03-29T10:39:36.181 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.197+0000 7f68d0ff9640 1 -- 172.21.15.179:0/1833069303 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e1569630-d46c-4cc7-93c7-73c7ad2140ac", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 715997913 0 940483276) 0x7f68c40056e0 con 0x7f68d406af60 2024-03-29T10:39:36.181 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.197+0000 7f68d0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e1569630-d46c-4cc7-93c7-73c7ad2140ac", "id": 1}] 2024-03-29T10:39:36.181 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.197+0000 7f68d0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:36.182 INFO:teuthology.orchestra.run.smithi179.stdout:1 2024-03-29T10:39:36.184 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.198+0000 7f68da35d640 10 librados: watch_flush enter 2024-03-29T10:39:36.184 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.198+0000 7f68da35d640 10 librados: watch_flush exit 2024-03-29T10:39:36.184 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.198+0000 7f68da35d640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-29T10:39:36.184 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.198+0000 7f68da35d640 10 client.4120.objecter successfully canceled tick 2024-03-29T10:39:36.184 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.198+0000 7f68da35d640 10 monclient: shutdown 2024-03-29T10:39:36.184 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.198+0000 7f68da35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:36.184 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.198+0000 7f68da35d640 1 -- 172.21.15.179:0/1833069303 >> v1:172.21.15.179:6789/0 conn(0x7f68d406af60 legacy=0x7f68d41d6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.184 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.198+0000 7f68da35d640 1 -- 172.21.15.179:0/1833069303 shutdown_connections 2024-03-29T10:39:36.185 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.198+0000 7f68da35d640 1 -- 172.21.15.179:0/1833069303 >> 172.21.15.179:0/1833069303 conn(0x7f68d40b58f0 msgr2=0x7f68d40b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:36.185 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.198+0000 7f68da35d640 1 -- 172.21.15.179:0/1833069303 shutdown_connections 2024-03-29T10:39:36.185 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.198+0000 7f68da35d640 1 -- 172.21.15.179:0/1833069303 wait complete. 2024-03-29T10:39:36.185 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.198+0000 7f68da35d640 1 librados: shutdown 2024-03-29T10:39:36.193 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 04c8ad6d-9d01-4655-80f4-334cb8e4f90e 2 2024-03-29T10:39:36.279 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:36.295+0000 7f4aa0845040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:39:36.293 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.309+0000 7f339224a640 10 monclient: get_monmap_and_config 2024-03-29T10:39:36.293 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.309+0000 7f339224a640 10 monclient: build_initial_monmap 2024-03-29T10:39:36.293 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.309+0000 7f339224a640 10 monclient: monmap: 2024-03-29T10:39:36.294 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T10:39:36.294 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.294 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:39:36.310336+0000 2024-03-29T10:39:36.294 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:39:36.310336+0000 2024-03-29T10:39:36.294 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:36.295 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.295 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:36.295 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:39:36.295 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:39:36.295 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.303 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.310+0000 7f339224a640 1 Processor -- start 2024-03-29T10:39:36.303 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.310+0000 7f339224a640 1 -- start start 2024-03-29T10:39:36.303 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.310+0000 7f339224a640 10 monclient: init 2024-03-29T10:39:36.303 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.310+0000 7f339224a640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:36.304 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.310+0000 7f339224a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:39:36.304 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.310+0000 7f339224a640 10 monclient(hunting): picked mon.noname-b con 0x7f338c09ac40 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:36.304 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.310+0000 7f339224a640 10 monclient(hunting): picked mon.noname-a con 0x7f338c09c870 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:36.304 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.310+0000 7f339224a640 10 monclient(hunting): picked mon.noname-c con 0x7f338c0b4360 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:36.304 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.310+0000 7f339224a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f338c09dec0 con 0x7f338c09c870 2024-03-29T10:39:36.304 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.310+0000 7f339224a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f338c09da30 con 0x7f338c0b4360 2024-03-29T10:39:36.304 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.310+0000 7f339224a640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f338c0b39c0 con 0x7f338c09ac40 2024-03-29T10:39:36.304 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.310+0000 7f339224a640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:36.305 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.310+0000 7f338bfff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f338c0b4360 0x7f338c14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:35126/0 (socket says 172.21.15.179:35126) 2024-03-29T10:39:36.305 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338bfff640 1 -- 172.21.15.179:0/2444541662 learned_addr learned my addr 172.21.15.179:0/2444541662 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T10:39:36.305 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f339224a640 10 monclient(hunting): authenticate will time out at 1088.580091s 2024-03-29T10:39:36.305 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 1 -- 172.21.15.179:0/2444541662 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814944930 0 0) 0x7f338c09da30 con 0x7f338c0b4360 2024-03-29T10:39:36.305 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.305 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.306 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.306 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 10 monclient(hunting): my global_id is 4123 2024-03-29T10:39:36.306 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 1 -- 172.21.15.179:0/2444541662 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3374003520 con 0x7f338c0b4360 2024-03-29T10:39:36.306 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 1 -- 172.21.15.179:0/2444541662 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117590538 0 0) 0x7f3380001040 con 0x7f338c09c870 2024-03-29T10:39:36.306 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.307 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.307 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.307 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 10 monclient(hunting): my global_id is 4122 2024-03-29T10:39:36.307 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 1 -- 172.21.15.179:0/2444541662 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f338c09da30 con 0x7f338c09c870 2024-03-29T10:39:36.307 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 1 -- 172.21.15.179:0/2444541662 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755402648 0 0) 0x7f338c0b39c0 con 0x7f338c09ac40 2024-03-29T10:39:36.307 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.307 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.308 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.308 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 10 monclient(hunting): my global_id is 4127 2024-03-29T10:39:36.308 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 1 -- 172.21.15.179:0/2444541662 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3380001040 con 0x7f338c09ac40 2024-03-29T10:39:36.308 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 1 -- 172.21.15.179:0/2444541662 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4032225798 0 0) 0x7f3374003520 con 0x7f338c0b4360 2024-03-29T10:39:36.308 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 1 -- 172.21.15.179:0/2444541662 >> v1:172.21.15.57:6790/0 conn(0x7f338c09ac40 legacy=0x7f338c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.308 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 1 -- 172.21.15.179:0/2444541662 >> v1:172.21.15.57:6789/0 conn(0x7f338c09c870 legacy=0x7f338c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.308 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:39:36.308 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 1 monclient: found mon.noname-c 2024-03-29T10:39:36.308 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:36.308 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.308 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 1 -- 172.21.15.179:0/2444541662 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338c0b5160 con 0x7f338c0b4360 2024-03-29T10:39:36.309 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:39:36.309 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:39:36.309 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:36.309 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 1 -- 172.21.15.179:0/2444541662 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f3384001a10 con 0x7f338c0b4360 2024-03-29T10:39:36.309 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f339224a640 5 monclient: authenticate success, global_id 4123 2024-03-29T10:39:36.309 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f339224a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:36.309 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.309 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:36.309 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 10 monclient: dump: 2024-03-29T10:39:36.309 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.309 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.309 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.309 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.309 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.309 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.310 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.310 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.310 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.310 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.310 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f339224a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:36.310 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 1 -- 172.21.15.179:0/2444541662 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3384001bf0 con 0x7f338c0b4360 2024-03-29T10:39:36.310 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:36.310 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f339224a640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:36.310 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f339224a640 4 set_mon_vals no callback set 2024-03-29T10:39:36.310 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 1 -- 172.21.15.179:0/2444541662 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f3384002040 con 0x7f338c0b4360 2024-03-29T10:39:36.310 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.310 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:36.311 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.311+0000 7f338a7fc640 10 monclient: dump: 2024-03-29T10:39:36.311 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.311 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.311 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.311 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.311 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.311 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.311 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.311 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.311 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.311 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.311 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.312+0000 7f339224a640 10 monclient: shutdown 2024-03-29T10:39:36.312 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.312+0000 7f339224a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:36.312 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.312+0000 7f339224a640 1 -- 172.21.15.179:0/2444541662 >> v1:172.21.15.179:6789/0 conn(0x7f338c0b4360 legacy=0x7f338c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.312 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.312+0000 7f339224a640 1 -- 172.21.15.179:0/2444541662 shutdown_connections 2024-03-29T10:39:36.312 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.312+0000 7f339224a640 1 -- 172.21.15.179:0/2444541662 >> 172.21.15.179:0/2444541662 conn(0x7f338c066b00 msgr2=0x7f338c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:36.312 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.312+0000 7f339224a640 1 -- 172.21.15.179:0/2444541662 shutdown_connections 2024-03-29T10:39:36.312 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.312+0000 7f339224a640 1 -- 172.21.15.179:0/2444541662 wait complete. 2024-03-29T10:39:36.312 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.312+0000 7f339224a640 10 monclient: build_initial_monmap 2024-03-29T10:39:36.312 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.312+0000 7f339224a640 10 monclient: monmap: 2024-03-29T10:39:36.312 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T10:39:36.312 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:36.312 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 0.000000 2024-03-29T10:39:36.312 INFO:teuthology.orchestra.run.smithi179.stderr:created 0.000000 2024-03-29T10:39:36.312 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:36.313 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.313 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:36.313 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:39:36.313 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:39:36.313 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.313 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.313+0000 7f339224a640 1 librados: starting msgr at 2024-03-29T10:39:36.313 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.313+0000 7f339224a640 1 librados: starting objecter 2024-03-29T10:39:36.313 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.313+0000 7f339224a640 1 Processor -- start 2024-03-29T10:39:36.313 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.313+0000 7f339224a640 1 -- start start 2024-03-29T10:39:36.313 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.313+0000 7f339224a640 1 librados: setting wanted keys 2024-03-29T10:39:36.313 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.313+0000 7f339224a640 1 librados: calling monclient init 2024-03-29T10:39:36.313 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.313+0000 7f339224a640 10 monclient: init 2024-03-29T10:39:36.313 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.313+0000 7f339224a640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:36.313 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.313+0000 7f339224a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:39:36.313 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.313+0000 7f339224a640 10 monclient(hunting): picked mon.noname-c con 0x7f338c09ac40 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:36.314 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.313+0000 7f339224a640 10 monclient(hunting): picked mon.noname-a con 0x7f338c140b60 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:36.314 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.313+0000 7f339224a640 10 monclient(hunting): picked mon.noname-b con 0x7f338c1d7fb0 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:36.314 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.313+0000 7f339224a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f338c142b50 con 0x7f338c140b60 2024-03-29T10:39:36.314 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f339224a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f338c1d7380 con 0x7f338c1d7fb0 2024-03-29T10:39:36.314 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f339224a640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f338c13ffa0 con 0x7f338c09ac40 2024-03-29T10:39:36.314 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f339224a640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:36.314 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f339224a640 10 monclient(hunting): authenticate will time out at 1088.583118s 2024-03-29T10:39:36.314 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f338bfff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f338c1d7fb0 0x7f338c136d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:35128/0 (socket says 172.21.15.179:35128) 2024-03-29T10:39:36.314 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f338bfff640 1 -- 172.21.15.179:0/96651493 learned_addr learned my addr 172.21.15.179:0/96651493 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T10:39:36.314 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 client.?.objecter ms_handle_connect 0x7f338c1d7fb0 2024-03-29T10:39:36.314 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:36.314 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 client.?.objecter ms_handle_connect 0x7f338c140b60 2024-03-29T10:39:36.315 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:36.315 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 client.?.objecter ms_handle_connect 0x7f338c09ac40 2024-03-29T10:39:36.315 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:36.315 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1331177124 0 0) 0x7f338c1d7380 con 0x7f338c1d7fb0 2024-03-29T10:39:36.315 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.315 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.315 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.315 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient(hunting): my global_id is 4126 2024-03-29T10:39:36.315 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3378004eb0 con 0x7f338c1d7fb0 2024-03-29T10:39:36.315 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 108646357 0 0) 0x7f338c142b50 con 0x7f338c140b60 2024-03-29T10:39:36.315 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.315 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.316 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.316 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient(hunting): my global_id is 4125 2024-03-29T10:39:36.316 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f338c1d7380 con 0x7f338c140b60 2024-03-29T10:39:36.316 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551886952 0 0) 0x7f338c13ffa0 con 0x7f338c09ac40 2024-03-29T10:39:36.316 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.316 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.316 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.316 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient(hunting): my global_id is 4130 2024-03-29T10:39:36.316 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f338c142b50 con 0x7f338c09ac40 2024-03-29T10:39:36.316 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4078388494 0 0) 0x7f3378004eb0 con 0x7f338c1d7fb0 2024-03-29T10:39:36.316 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f338c13ffa0 con 0x7f338c1d7fb0 2024-03-29T10:39:36.316 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f3384001f40 con 0x7f338c1d7fb0 2024-03-29T10:39:36.316 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.316 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:36.317 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient(hunting): dump: 2024-03-29T10:39:36.317 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.317 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.317 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.317 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.317 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.317 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.317 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.317 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.317 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.317 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.317 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:36.317 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 908046512 0 0) 0x7f338c1d7380 con 0x7f338c140b60 2024-03-29T10:39:36.317 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3378004eb0 con 0x7f338c140b60 2024-03-29T10:39:36.317 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f3380001a50 con 0x7f338c140b60 2024-03-29T10:39:36.317 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.318 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:36.318 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient(hunting): dump: 2024-03-29T10:39:36.318 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.318 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.318 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.318 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.318 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.318 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.318 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.318 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.318 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.318 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.318 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3787350987 0 0) 0x7f338c142b50 con 0x7f338c09ac40 2024-03-29T10:39:36.319 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f338c1d7380 con 0x7f338c09ac40 2024-03-29T10:39:36.319 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f337c001830 con 0x7f338c09ac40 2024-03-29T10:39:36.319 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.319 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:36.319 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient(hunting): dump: 2024-03-29T10:39:36.319 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.319 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.319 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.319 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.319 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.319 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.319 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.319 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.319 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.319 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.320 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2153118004 0 0) 0x7f338c13ffa0 con 0x7f338c1d7fb0 2024-03-29T10:39:36.320 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 >> v1:172.21.15.57:6790/0 conn(0x7f338c09ac40 legacy=0x7f338c1417c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.320 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 >> v1:172.21.15.57:6789/0 conn(0x7f338c140b60 legacy=0x7f338c1d78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.320 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient: _finish_hunting 0 2024-03-29T10:39:36.320 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 1 monclient: found mon.b 2024-03-29T10:39:36.320 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:36.320 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.320 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338c144560 con 0x7f338c1d7fb0 2024-03-29T10:39:36.320 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient: _finish_auth 0 2024-03-29T10:39:36.320 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 10 monclient: _check_auth_tickets 2024-03-29T10:39:36.320 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f3373fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:36.320 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f339224a640 5 monclient: authenticate success, global_id 4126 2024-03-29T10:39:36.320 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f339224a640 10 monclient: _renew_subs 2024-03-29T10:39:36.320 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f339224a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.321 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f339224a640 1 -- 172.21.15.179:0/96651493 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f338c137490 con 0x7f338c1d7fb0 2024-03-29T10:39:36.321 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f339224a640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:36.321 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f339224a640 10 monclient: _renew_subs 2024-03-29T10:39:36.321 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f339224a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.321 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f339224a640 1 -- 172.21.15.179:0/96651493 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f338c1e3c10 con 0x7f338c1d7fb0 2024-03-29T10:39:36.321 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.314+0000 7f339224a640 1 librados: init done 2024-03-29T10:39:36.321 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.315+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3384004e10 con 0x7f338c1d7fb0 2024-03-29T10:39:36.321 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.315+0000 7f3373fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:36.321 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.315+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f3384005120 con 0x7f338c1d7fb0 2024-03-29T10:39:36.321 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.315+0000 7f3373fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.322 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.315+0000 7f3373fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:36.322 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.315+0000 7f3389ffb640 4 set_mon_vals no callback set 2024-03-29T10:39:36.322 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.315+0000 7f3373fff640 10 monclient: dump: 2024-03-29T10:39:36.322 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.322 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.322 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.322 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.322 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.322 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.323 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.323 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.323 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.323 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.323 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.315+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3384005610 con 0x7f338c1d7fb0 2024-03-29T10:39:36.323 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.315+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 3699475579 0 0) 0x7f3384006a90 con 0x7f338c1d7fb0 2024-03-29T10:39:36.323 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.315+0000 7f3373fff640 10 client.4126.objecter ms_dispatch 0x7f338c000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T10:39:36.323 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.315+0000 7f3373fff640 3 client.4126.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T10:39:36.323 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.315+0000 7f3373fff640 3 client.4126.objecter handle_osd_map decoding full epoch 4 2024-03-29T10:39:36.323 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.315+0000 7f3373fff640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-29T10:39:36.323 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.315+0000 7f339224a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:36.323 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.315+0000 7f339224a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:36.323 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.315+0000 7f339224a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.323 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.315+0000 7f339224a640 1 -- 172.21.15.179:0/96651493 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f336c005080 con 0x7f338c1d7fb0 2024-03-29T10:39:36.324 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.317+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f33840066e0 con 0x7f338c1d7fb0 2024-03-29T10:39:36.324 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.317+0000 7f3373fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:36.324 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.317+0000 7f3373fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:36.324 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:36.328+0000 7f8972a32040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:39:36.337 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:36.353+0000 7f4aa0845040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:39:36.368 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:36.384+0000 7f8972a32040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:39:36.373 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.389+0000 7f339224a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "04c8ad6d-9d01-4655-80f4-334cb8e4f90e", "id": 2}] 2024-03-29T10:39:36.373 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.389+0000 7f339224a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "04c8ad6d-9d01-4655-80f4-334cb8e4f90e", "id": 2}] 2024-03-29T10:39:36.373 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.389+0000 7f339224a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.373 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.389+0000 7f339224a640 1 -- 172.21.15.179:0/96651493 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd new", "uuid": "04c8ad6d-9d01-4655-80f4-334cb8e4f90e", "id": 2} v 0) v1 -- 0x7f336c0058b0 con 0x7f338c1d7fb0 2024-03-29T10:39:36.382 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.398+0000 7f3373fff640 1 -- 172.21.15.179:0/96651493 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "04c8ad6d-9d01-4655-80f4-334cb8e4f90e", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3393693995 0 216599125) 0x7f3384005a00 con 0x7f338c1d7fb0 2024-03-29T10:39:36.382 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.398+0000 7f3373fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "04c8ad6d-9d01-4655-80f4-334cb8e4f90e", "id": 2}] 2024-03-29T10:39:36.382 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.398+0000 7f3373fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:36.383 INFO:teuthology.orchestra.run.smithi179.stdout:2 2024-03-29T10:39:36.383 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:36.399+0000 7f4aa0845040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:39:36.384 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.399+0000 7f339224a640 10 librados: watch_flush enter 2024-03-29T10:39:36.384 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.399+0000 7f339224a640 10 librados: watch_flush exit 2024-03-29T10:39:36.384 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.399+0000 7f339224a640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-29T10:39:36.384 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.399+0000 7f339224a640 10 client.4126.objecter successfully canceled tick 2024-03-29T10:39:36.384 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.399+0000 7f339224a640 10 monclient: shutdown 2024-03-29T10:39:36.384 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.399+0000 7f339224a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:36.384 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.399+0000 7f339224a640 1 -- 172.21.15.179:0/96651493 >> v1:172.21.15.179:6789/0 conn(0x7f338c1d7fb0 legacy=0x7f338c136d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.384 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.399+0000 7f339224a640 1 -- 172.21.15.179:0/96651493 shutdown_connections 2024-03-29T10:39:36.384 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.399+0000 7f339224a640 1 -- 172.21.15.179:0/96651493 >> 172.21.15.179:0/96651493 conn(0x7f338c066b00 msgr2=0x7f338c13d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:36.384 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.400+0000 7f339224a640 1 -- 172.21.15.179:0/96651493 shutdown_connections 2024-03-29T10:39:36.385 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.400+0000 7f339224a640 1 -- 172.21.15.179:0/96651493 wait complete. 2024-03-29T10:39:36.385 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.400+0000 7f339224a640 1 librados: shutdown 2024-03-29T10:39:36.394 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 5566b3b8-270a-45b8-a6ba-f5f0228937e5 3 2024-03-29T10:39:36.415 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:36.431+0000 7f8972a32040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:39:36.441 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:36.457+0000 7f4aa0845040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:39:36.470 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:36.487+0000 7f8972a32040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:39:36.486 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:36.501+0000 7f4aa0845040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:39:36.494 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.510+0000 7ff27f863640 10 monclient: get_monmap_and_config 2024-03-29T10:39:36.494 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.510+0000 7ff27f863640 10 monclient: build_initial_monmap 2024-03-29T10:39:36.494 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.510+0000 7ff27f863640 10 monclient: monmap: 2024-03-29T10:39:36.494 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T10:39:36.494 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.494 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:39:36.511253+0000 2024-03-29T10:39:36.495 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:39:36.511253+0000 2024-03-29T10:39:36.495 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:36.495 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.495 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:36.495 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:39:36.495 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:39:36.495 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.505 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.511+0000 7ff27f863640 1 Processor -- start 2024-03-29T10:39:36.505 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.511+0000 7ff27f863640 1 -- start start 2024-03-29T10:39:36.506 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.511+0000 7ff27f863640 10 monclient: init 2024-03-29T10:39:36.506 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.511+0000 7ff27f863640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:36.506 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.511+0000 7ff27f863640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:39:36.506 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff27f863640 10 monclient(hunting): picked mon.noname-b con 0x7ff27809ac40 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:36.506 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff27f863640 10 monclient(hunting): picked mon.noname-c con 0x7ff27809c870 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:36.506 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff27f863640 10 monclient(hunting): picked mon.noname-a con 0x7ff2780b4360 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:36.506 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff27f863640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff27809dec0 con 0x7ff2780b4360 2024-03-29T10:39:36.506 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff27f863640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff27809da30 con 0x7ff27809c870 2024-03-29T10:39:36.506 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff27f863640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2780b39c0 con 0x7ff27809ac40 2024-03-29T10:39:36.506 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff27f863640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:36.506 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff27f863640 10 monclient(hunting): authenticate will time out at 1088.781180s 2024-03-29T10:39:36.506 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff27cdfe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7ff27809c870 0x7ff27813f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:35138/0 (socket says 172.21.15.179:35138) 2024-03-29T10:39:36.506 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff27cdfe640 1 -- 172.21.15.179:0/3347162645 learned_addr learned my addr 172.21.15.179:0/3347162645 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T10:39:36.507 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 1 -- 172.21.15.179:0/3347162645 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598591122 0 0) 0x7ff27809da30 con 0x7ff27809c870 2024-03-29T10:39:36.507 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.507 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.507 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.507 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 10 monclient(hunting): my global_id is 4129 2024-03-29T10:39:36.507 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 1 -- 172.21.15.179:0/3347162645 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff264003520 con 0x7ff27809c870 2024-03-29T10:39:36.507 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 1 -- 172.21.15.179:0/3347162645 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909584660 0 0) 0x7ff2780b39c0 con 0x7ff27809ac40 2024-03-29T10:39:36.507 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.507 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.507 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.507 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 10 monclient(hunting): my global_id is 4133 2024-03-29T10:39:36.507 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 1 -- 172.21.15.179:0/3347162645 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff27809da30 con 0x7ff27809ac40 2024-03-29T10:39:36.507 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 1 -- 172.21.15.179:0/3347162645 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2093309478 0 0) 0x7ff2700013c0 con 0x7ff2780b4360 2024-03-29T10:39:36.507 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.507 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 10 monclient(hunting): my global_id is 4128 2024-03-29T10:39:36.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 1 -- 172.21.15.179:0/3347162645 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2780b39c0 con 0x7ff2780b4360 2024-03-29T10:39:36.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 1 -- 172.21.15.179:0/3347162645 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4255822368 0 0) 0x7ff264003520 con 0x7ff27809c870 2024-03-29T10:39:36.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 1 -- 172.21.15.179:0/3347162645 >> v1:172.21.15.57:6790/0 conn(0x7ff27809ac40 legacy=0x7ff2780b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 1 -- 172.21.15.179:0/3347162645 >> v1:172.21.15.57:6789/0 conn(0x7ff2780b4360 legacy=0x7ff27814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:39:36.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 1 monclient: found mon.noname-c 2024-03-29T10:39:36.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:36.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 1 -- 172.21.15.179:0/3347162645 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2780b5160 con 0x7ff27809c870 2024-03-29T10:39:36.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 10 monclient: _finish_auth 0 2024-03-29T10:39:36.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:39:36.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:36.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 1 -- 172.21.15.179:0/3347162645 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7ff268001690 con 0x7ff27809c870 2024-03-29T10:39:36.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff27f863640 5 monclient: authenticate success, global_id 4129 2024-03-29T10:39:36.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:36.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 10 monclient: dump: 2024-03-29T10:39:36.509 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.509 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.509 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.509 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.509 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.509 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.509 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.509 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.509 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.509 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff27f863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:36.510 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 1 -- 172.21.15.179:0/3347162645 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff268001870 con 0x7ff27809c870 2024-03-29T10:39:36.510 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:36.510 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff27f863640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:36.510 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 1 -- 172.21.15.179:0/3347162645 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7ff268001c80 con 0x7ff27809c870 2024-03-29T10:39:36.510 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.510 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff27f863640 4 set_mon_vals no callback set 2024-03-29T10:39:36.510 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:36.510 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.512+0000 7ff26ffff640 10 monclient: dump: 2024-03-29T10:39:36.510 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.510 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.510 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.510 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.510 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.510 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.510 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.511 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.511 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.511 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.511 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.513+0000 7ff27f863640 10 monclient: shutdown 2024-03-29T10:39:36.511 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.513+0000 7ff27f863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:36.511 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.513+0000 7ff27f863640 1 -- 172.21.15.179:0/3347162645 >> v1:172.21.15.179:6789/0 conn(0x7ff27809c870 legacy=0x7ff27813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.511 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.513+0000 7ff27f863640 1 -- 172.21.15.179:0/3347162645 shutdown_connections 2024-03-29T10:39:36.511 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.513+0000 7ff27f863640 1 -- 172.21.15.179:0/3347162645 >> 172.21.15.179:0/3347162645 conn(0x7ff278066b00 msgr2=0x7ff278066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:36.511 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.513+0000 7ff27f863640 1 -- 172.21.15.179:0/3347162645 shutdown_connections 2024-03-29T10:39:36.511 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.513+0000 7ff27f863640 1 -- 172.21.15.179:0/3347162645 wait complete. 2024-03-29T10:39:36.511 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.514+0000 7ff27f863640 10 monclient: build_initial_monmap 2024-03-29T10:39:36.511 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.514+0000 7ff27f863640 10 monclient: monmap: 2024-03-29T10:39:36.511 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T10:39:36.511 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:36.511 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 0.000000 2024-03-29T10:39:36.512 INFO:teuthology.orchestra.run.smithi179.stderr:created 0.000000 2024-03-29T10:39:36.512 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:36.512 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.512 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:36.512 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:39:36.512 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:39:36.512 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.512 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.514+0000 7ff27f863640 1 librados: starting msgr at 2024-03-29T10:39:36.512 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.514+0000 7ff27f863640 1 librados: starting objecter 2024-03-29T10:39:36.512 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.514+0000 7ff27f863640 1 Processor -- start 2024-03-29T10:39:36.512 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.514+0000 7ff27f863640 1 -- start start 2024-03-29T10:39:36.512 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.514+0000 7ff27f863640 1 librados: setting wanted keys 2024-03-29T10:39:36.512 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.514+0000 7ff27f863640 1 librados: calling monclient init 2024-03-29T10:39:36.512 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.514+0000 7ff27f863640 10 monclient: init 2024-03-29T10:39:36.512 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff27f863640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:36.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff27f863640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:39:36.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff27f863640 10 monclient(hunting): picked mon.noname-b con 0x7ff27809ac40 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:36.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff27f863640 10 monclient(hunting): picked mon.noname-a con 0x7ff278138f60 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:36.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff27f863640 10 monclient(hunting): picked mon.noname-c con 0x7ff2780afe50 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:36.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff27f863640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2781db670 con 0x7ff278138f60 2024-03-29T10:39:36.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff27f863640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2781db300 con 0x7ff27809ac40 2024-03-29T10:39:36.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff27f863640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2780b1aa0 con 0x7ff2780afe50 2024-03-29T10:39:36.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff27f863640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:36.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff27f863640 10 monclient(hunting): authenticate will time out at 1088.784346s 2024-03-29T10:39:36.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff27d5ff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7ff27809ac40 0x7ff2781dabf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:35144/0 (socket says 172.21.15.179:35144) 2024-03-29T10:39:36.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff27d5ff640 1 -- 172.21.15.179:0/3178973292 learned_addr learned my addr 172.21.15.179:0/3178973292 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T10:39:36.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 client.?.objecter ms_handle_connect 0x7ff27809ac40 2024-03-29T10:39:36.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:36.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 client.?.objecter ms_handle_connect 0x7ff278138f60 2024-03-29T10:39:36.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:36.514 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 client.?.objecter ms_handle_connect 0x7ff2780afe50 2024-03-29T10:39:36.514 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:36.514 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787156831 0 0) 0x7ff2781db300 con 0x7ff27809ac40 2024-03-29T10:39:36.514 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.514 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.514 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.514 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 monclient(hunting): my global_id is 4132 2024-03-29T10:39:36.514 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff25c004eb0 con 0x7ff27809ac40 2024-03-29T10:39:36.514 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058972968 0 0) 0x7ff2781db670 con 0x7ff278138f60 2024-03-29T10:39:36.514 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.514 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.514 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.514 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 monclient(hunting): my global_id is 4131 2024-03-29T10:39:36.514 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2781db300 con 0x7ff278138f60 2024-03-29T10:39:36.515 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610748014 0 0) 0x7ff2780b1aa0 con 0x7ff2780afe50 2024-03-29T10:39:36.515 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.515 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.515 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.515 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 monclient(hunting): my global_id is 4136 2024-03-29T10:39:36.515 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2781db670 con 0x7ff2780afe50 2024-03-29T10:39:36.515 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2981123409 0 0) 0x7ff25c004eb0 con 0x7ff27809ac40 2024-03-29T10:39:36.515 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2780b1aa0 con 0x7ff27809ac40 2024-03-29T10:39:36.515 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7ff274001a30 con 0x7ff27809ac40 2024-03-29T10:39:36.515 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.515 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:36.515 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 monclient(hunting): dump: 2024-03-29T10:39:36.516 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.516 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.516 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.516 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.516 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.516 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.516 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.516 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.516 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.516 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.516 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:36.516 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1559102317 0 0) 0x7ff2781db300 con 0x7ff278138f60 2024-03-29T10:39:36.516 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff25c004eb0 con 0x7ff278138f60 2024-03-29T10:39:36.516 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7ff268001b80 con 0x7ff278138f60 2024-03-29T10:39:36.516 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.517 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:36.517 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 monclient(hunting): dump: 2024-03-29T10:39:36.517 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.517 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.517 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.517 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.517 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.517 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.517 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.517 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.517 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.517 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.517 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1516526944 0 0) 0x7ff2781db670 con 0x7ff2780afe50 2024-03-29T10:39:36.517 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2781db300 con 0x7ff2780afe50 2024-03-29T10:39:36.517 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7ff2700017a0 con 0x7ff2780afe50 2024-03-29T10:39:36.518 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.518 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:36.518 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 10 monclient(hunting): dump: 2024-03-29T10:39:36.518 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.518 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.518 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.518 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.518 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.518 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.518 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.518 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.518 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.518 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.519 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.515+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2633102201 0 0) 0x7ff2780b1aa0 con 0x7ff27809ac40 2024-03-29T10:39:36.519 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 >> v1:172.21.15.57:6790/0 conn(0x7ff2780afe50 legacy=0x7ff2781e1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.519 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 >> v1:172.21.15.57:6789/0 conn(0x7ff278138f60 legacy=0x7ff2780af740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.519 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff26dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:39:36.519 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff26dffb640 1 monclient: found mon.b 2024-03-29T10:39:36.519 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff26dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:36.519 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff26dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.519 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2780b02a0 con 0x7ff27809ac40 2024-03-29T10:39:36.519 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff26dffb640 10 monclient: _finish_auth 0 2024-03-29T10:39:36.519 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff26dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:39:36.519 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff26dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:36.519 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff27f863640 5 monclient: authenticate success, global_id 4132 2024-03-29T10:39:36.519 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff27f863640 10 monclient: _renew_subs 2024-03-29T10:39:36.519 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff27f863640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.519 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff27f863640 1 -- 172.21.15.179:0/3178973292 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2781e35e0 con 0x7ff27809ac40 2024-03-29T10:39:36.520 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff274001cf0 con 0x7ff27809ac40 2024-03-29T10:39:36.520 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff26dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:36.520 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7ff274003a60 con 0x7ff27809ac40 2024-03-29T10:39:36.520 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff26dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.520 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff26dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:36.520 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff26dffb640 10 monclient: dump: 2024-03-29T10:39:36.520 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.520 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.520 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.520 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.520 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.520 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.520 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.520 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.521 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.521 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.521 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff26f7fe640 4 set_mon_vals no callback set 2024-03-29T10:39:36.521 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff274004010 con 0x7ff27809ac40 2024-03-29T10:39:36.521 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff27f863640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:36.521 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff27f863640 10 monclient: _renew_subs 2024-03-29T10:39:36.521 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff27f863640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.521 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff27f863640 1 -- 172.21.15.179:0/3178973292 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2781e3950 con 0x7ff27809ac40 2024-03-29T10:39:36.521 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff27f863640 1 librados: init done 2024-03-29T10:39:36.521 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3023741673 0 0) 0x7ff274005c50 con 0x7ff27809ac40 2024-03-29T10:39:36.521 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff26dffb640 10 client.4132.objecter ms_dispatch 0x7ff278000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T10:39:36.521 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff26dffb640 3 client.4132.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T10:39:36.521 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff26dffb640 3 client.4132.objecter handle_osd_map decoding full epoch 5 2024-03-29T10:39:36.521 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.516+0000 7ff26dffb640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-29T10:39:36.521 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.517+0000 7ff27f863640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:36.522 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.517+0000 7ff27f863640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:36.522 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.517+0000 7ff27f863640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.522 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.517+0000 7ff27f863640 1 -- 172.21.15.179:0/3178973292 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff27809b020 con 0x7ff27809ac40 2024-03-29T10:39:36.522 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.519+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff2740057c0 con 0x7ff27809ac40 2024-03-29T10:39:36.522 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.519+0000 7ff26dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:36.522 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.519+0000 7ff26dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:36.522 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:36.531+0000 7f8972a32040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:39:36.589 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.605+0000 7ff27f863640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5566b3b8-270a-45b8-a6ba-f5f0228937e5", "id": 3}] 2024-03-29T10:39:36.589 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.605+0000 7ff27f863640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5566b3b8-270a-45b8-a6ba-f5f0228937e5", "id": 3}] 2024-03-29T10:39:36.589 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.605+0000 7ff27f863640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.589 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.605+0000 7ff27f863640 1 -- 172.21.15.179:0/3178973292 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd new", "uuid": "5566b3b8-270a-45b8-a6ba-f5f0228937e5", "id": 3} v 0) v1 -- 0x7ff27813e470 con 0x7ff27809ac40 2024-03-29T10:39:36.595 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.611+0000 7ff26dffb640 1 -- 172.21.15.179:0/3178973292 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5566b3b8-270a-45b8-a6ba-f5f0228937e5", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 4063198279 0 525046306) 0x7ff274005aa0 con 0x7ff27809ac40 2024-03-29T10:39:36.595 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.611+0000 7ff26dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5566b3b8-270a-45b8-a6ba-f5f0228937e5", "id": 3}] 2024-03-29T10:39:36.595 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.611+0000 7ff26dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:36.595 INFO:teuthology.orchestra.run.smithi179.stdout:3 2024-03-29T10:39:36.596 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.612+0000 7ff27f863640 10 librados: watch_flush enter 2024-03-29T10:39:36.597 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.612+0000 7ff27f863640 10 librados: watch_flush exit 2024-03-29T10:39:36.597 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.612+0000 7ff27f863640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-29T10:39:36.597 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.612+0000 7ff27f863640 10 client.4132.objecter successfully canceled tick 2024-03-29T10:39:36.597 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.612+0000 7ff27f863640 10 monclient: shutdown 2024-03-29T10:39:36.597 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.612+0000 7ff27f863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:36.597 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.612+0000 7ff27f863640 1 -- 172.21.15.179:0/3178973292 >> v1:172.21.15.179:6789/0 conn(0x7ff27809ac40 legacy=0x7ff2781dabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.597 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.612+0000 7ff27f863640 1 -- 172.21.15.179:0/3178973292 shutdown_connections 2024-03-29T10:39:36.597 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.612+0000 7ff27f863640 1 -- 172.21.15.179:0/3178973292 >> 172.21.15.179:0/3178973292 conn(0x7ff278066b00 msgr2=0x7ff27813d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:36.597 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.612+0000 7ff27f863640 1 -- 172.21.15.179:0/3178973292 shutdown_connections 2024-03-29T10:39:36.597 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.612+0000 7ff27f863640 1 -- 172.21.15.179:0/3178973292 wait complete. 2024-03-29T10:39:36.597 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.612+0000 7ff27f863640 1 librados: shutdown 2024-03-29T10:39:36.606 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new a853ea6e-33dd-42f3-a7a9-0d424e82f27d 4 2024-03-29T10:39:36.620 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:36.636+0000 7f4aa0845040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:39:36.635 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:36.651+0000 7f8972a32040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:39:36.694 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:36.710+0000 7f4aa0845040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:39:36.707 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.723+0000 7f3db6ed9640 10 monclient: get_monmap_and_config 2024-03-29T10:39:36.707 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.723+0000 7f3db6ed9640 10 monclient: build_initial_monmap 2024-03-29T10:39:36.707 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.723+0000 7f3db6ed9640 10 monclient: monmap: 2024-03-29T10:39:36.707 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T10:39:36.707 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.707 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:39:36.724224+0000 2024-03-29T10:39:36.707 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:39:36.724224+0000 2024-03-29T10:39:36.707 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:36.707 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.708 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:36.708 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:39:36.708 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:39:36.708 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.709 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:36.724+0000 7f8972a32040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:39:36.718 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.723+0000 7f3db6ed9640 1 Processor -- start 2024-03-29T10:39:36.718 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.724+0000 7f3db6ed9640 1 -- start start 2024-03-29T10:39:36.718 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.724+0000 7f3db6ed9640 10 monclient: init 2024-03-29T10:39:36.718 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.724+0000 7f3db6ed9640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:36.718 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.724+0000 7f3db6ed9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:36.719 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.724+0000 7f3db6ed9640 10 monclient(hunting): picked mon.noname-c con 0x7f3db009ac40 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:36.719 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.724+0000 7f3db6ed9640 10 monclient(hunting): picked mon.noname-b con 0x7f3db009c870 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:36.719 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.724+0000 7f3db6ed9640 10 monclient(hunting): picked mon.noname-a con 0x7f3db00b4360 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:36.719 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.724+0000 7f3db6ed9640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db009dec0 con 0x7f3db00b4360 2024-03-29T10:39:36.719 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.724+0000 7f3db6ed9640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db009da30 con 0x7f3db009ac40 2024-03-29T10:39:36.719 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.724+0000 7f3db6ed9640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db00b39c0 con 0x7f3db009c870 2024-03-29T10:39:36.719 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.724+0000 7f3db6ed9640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:36.719 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.724+0000 7f3db6ed9640 10 monclient(hunting): authenticate will time out at 1088.993761s 2024-03-29T10:39:36.719 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.724+0000 7f3db4c75640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f3db009ac40 0x7f3db00b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:35152/0 (socket says 172.21.15.179:35152) 2024-03-29T10:39:36.719 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.724+0000 7f3db4c75640 1 -- 172.21.15.179:0/1426363249 learned_addr learned my addr 172.21.15.179:0/1426363249 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T10:39:36.719 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.724+0000 7f3daf7fe640 1 -- 172.21.15.179:0/1426363249 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1330765570 0 0) 0x7f3db009da30 con 0x7f3db009ac40 2024-03-29T10:39:36.719 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.720 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.720 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.720 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 10 monclient(hunting): my global_id is 4135 2024-03-29T10:39:36.720 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 1 -- 172.21.15.179:0/1426363249 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3da0003520 con 0x7f3db009ac40 2024-03-29T10:39:36.720 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 1 -- 172.21.15.179:0/1426363249 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2833392260 0 0) 0x7f3da8001490 con 0x7f3db00b4360 2024-03-29T10:39:36.720 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.720 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.720 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.720 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 10 monclient(hunting): my global_id is 4134 2024-03-29T10:39:36.720 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 1 -- 172.21.15.179:0/1426363249 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3db009da30 con 0x7f3db00b4360 2024-03-29T10:39:36.720 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 1 -- 172.21.15.179:0/1426363249 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1110849658 0 0) 0x7f3db00b39c0 con 0x7f3db009c870 2024-03-29T10:39:36.720 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.720 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.720 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.720 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 10 monclient(hunting): my global_id is 4139 2024-03-29T10:39:36.721 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 1 -- 172.21.15.179:0/1426363249 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3da8001490 con 0x7f3db009c870 2024-03-29T10:39:36.721 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 1 -- 172.21.15.179:0/1426363249 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3193017621 0 0) 0x7f3da0003520 con 0x7f3db009ac40 2024-03-29T10:39:36.721 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 1 -- 172.21.15.179:0/1426363249 >> v1:172.21.15.57:6790/0 conn(0x7f3db009c870 legacy=0x7f3db013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.721 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 1 -- 172.21.15.179:0/1426363249 >> v1:172.21.15.57:6789/0 conn(0x7f3db00b4360 legacy=0x7f3db014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.721 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:39:36.721 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 1 monclient: found mon.noname-c 2024-03-29T10:39:36.721 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:36.721 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.721 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 1 -- 172.21.15.179:0/1426363249 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db00b5160 con 0x7f3db009ac40 2024-03-29T10:39:36.721 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 10 monclient: _finish_auth 0 2024-03-29T10:39:36.721 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:39:36.721 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:36.721 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3db6ed9640 5 monclient: authenticate success, global_id 4135 2024-03-29T10:39:36.721 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3db6ed9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:36.722 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 1 -- 172.21.15.179:0/1426363249 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f3da4001690 con 0x7f3db009ac40 2024-03-29T10:39:36.722 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.722 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:36.722 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 10 monclient: dump: 2024-03-29T10:39:36.722 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.722 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.722 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.722 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.722 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.722 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.722 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.722 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.722 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.722 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.723 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3db6ed9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:36.723 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 1 -- 172.21.15.179:0/1426363249 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3da4001870 con 0x7f3db009ac40 2024-03-29T10:39:36.723 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:36.723 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3db6ed9640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:36.723 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 1 -- 172.21.15.179:0/1426363249 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f3da4001c80 con 0x7f3db009ac40 2024-03-29T10:39:36.723 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3db6ed9640 4 set_mon_vals no callback set 2024-03-29T10:39:36.723 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.723 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:36.723 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3daf7fe640 10 monclient: dump: 2024-03-29T10:39:36.723 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.723 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.723 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.723 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.723 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.724 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.724 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.724 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.724 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.724 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.724 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3db6ed9640 10 monclient: shutdown 2024-03-29T10:39:36.724 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3db6ed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:36.724 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3db6ed9640 1 -- 172.21.15.179:0/1426363249 >> v1:172.21.15.179:6789/0 conn(0x7f3db009ac40 legacy=0x7f3db00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.724 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3db6ed9640 1 -- 172.21.15.179:0/1426363249 shutdown_connections 2024-03-29T10:39:36.724 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.725+0000 7f3db6ed9640 1 -- 172.21.15.179:0/1426363249 >> 172.21.15.179:0/1426363249 conn(0x7f3db0066b00 msgr2=0x7f3db0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:36.724 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.726+0000 7f3db6ed9640 1 -- 172.21.15.179:0/1426363249 shutdown_connections 2024-03-29T10:39:36.725 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.726+0000 7f3db6ed9640 1 -- 172.21.15.179:0/1426363249 wait complete. 2024-03-29T10:39:36.725 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.726+0000 7f3db6ed9640 10 monclient: build_initial_monmap 2024-03-29T10:39:36.725 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.726+0000 7f3db6ed9640 10 monclient: monmap: 2024-03-29T10:39:36.725 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T10:39:36.725 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:36.725 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 0.000000 2024-03-29T10:39:36.725 INFO:teuthology.orchestra.run.smithi179.stderr:created 0.000000 2024-03-29T10:39:36.725 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:36.725 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.725 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:36.725 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:39:36.725 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:39:36.725 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.725 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.726+0000 7f3db6ed9640 1 librados: starting msgr at 2024-03-29T10:39:36.725 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.726+0000 7f3db6ed9640 1 librados: starting objecter 2024-03-29T10:39:36.726 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.726+0000 7f3db6ed9640 1 Processor -- start 2024-03-29T10:39:36.726 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.726+0000 7f3db6ed9640 1 -- start start 2024-03-29T10:39:36.726 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.726+0000 7f3db6ed9640 1 librados: setting wanted keys 2024-03-29T10:39:36.726 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.726+0000 7f3db6ed9640 1 librados: calling monclient init 2024-03-29T10:39:36.726 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.727+0000 7f3db6ed9640 10 monclient: init 2024-03-29T10:39:36.726 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.727+0000 7f3db6ed9640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:36.726 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.727+0000 7f3db6ed9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:39:36.726 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.727+0000 7f3db6ed9640 10 monclient(hunting): picked mon.noname-a con 0x7f3db009ac40 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:36.726 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.727+0000 7f3db6ed9640 10 monclient(hunting): picked mon.noname-c con 0x7f3db00b12b0 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:36.726 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.727+0000 7f3db6ed9640 10 monclient(hunting): picked mon.noname-b con 0x7f3db01d7a40 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:36.726 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.727+0000 7f3db6ed9640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db00b8000 con 0x7f3db009ac40 2024-03-29T10:39:36.726 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.727+0000 7f3db6ed9640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db0142160 con 0x7f3db01d7a40 2024-03-29T10:39:36.726 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.727+0000 7f3db6ed9640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db0142f70 con 0x7f3db00b12b0 2024-03-29T10:39:36.726 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.727+0000 7f3db6ed9640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:36.727 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.727+0000 7f3db6ed9640 10 monclient(hunting): authenticate will time out at 1088.996798s 2024-03-29T10:39:36.727 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.727+0000 7f3db5476640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f3db01d7a40 0x7f3db01e1ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:35162/0 (socket says 172.21.15.179:35162) 2024-03-29T10:39:36.727 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.727+0000 7f3db5476640 1 -- 172.21.15.179:0/1657312323 learned_addr learned my addr 172.21.15.179:0/1657312323 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T10:39:36.727 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.727+0000 7f3dad7fa640 10 client.?.objecter ms_handle_connect 0x7f3db01d7a40 2024-03-29T10:39:36.727 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.727+0000 7f3dad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:36.727 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.727+0000 7f3dad7fa640 10 client.?.objecter ms_handle_connect 0x7f3db009ac40 2024-03-29T10:39:36.727 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.727+0000 7f3dad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:36.727 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.727+0000 7f3dad7fa640 10 client.?.objecter ms_handle_connect 0x7f3db00b12b0 2024-03-29T10:39:36.727 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.727+0000 7f3dad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:36.727 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.727+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 127049298 0 0) 0x7f3db0142160 con 0x7f3db01d7a40 2024-03-29T10:39:36.727 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.727 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.727 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.727 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient(hunting): my global_id is 4138 2024-03-29T10:39:36.728 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d94004eb0 con 0x7f3db01d7a40 2024-03-29T10:39:36.728 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 943017605 0 0) 0x7f3db00b8000 con 0x7f3db009ac40 2024-03-29T10:39:36.728 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.728 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.728 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.728 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient(hunting): my global_id is 4137 2024-03-29T10:39:36.728 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3db0142160 con 0x7f3db009ac40 2024-03-29T10:39:36.728 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761967980 0 0) 0x7f3db0142f70 con 0x7f3db00b12b0 2024-03-29T10:39:36.728 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.728 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.728 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.728 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient(hunting): my global_id is 4142 2024-03-29T10:39:36.728 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3db00b8000 con 0x7f3db00b12b0 2024-03-29T10:39:36.728 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2206717974 0 0) 0x7f3d94004eb0 con 0x7f3db01d7a40 2024-03-29T10:39:36.728 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3db0142f70 con 0x7f3db01d7a40 2024-03-29T10:39:36.729 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f3da8001bf0 con 0x7f3db01d7a40 2024-03-29T10:39:36.729 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.729 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:36.729 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient(hunting): dump: 2024-03-29T10:39:36.729 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.729 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.729 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.729 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.729 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.729 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.729 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.729 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.729 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.729 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.729 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:36.730 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 251233740 0 0) 0x7f3db0142160 con 0x7f3db009ac40 2024-03-29T10:39:36.730 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d94004eb0 con 0x7f3db009ac40 2024-03-29T10:39:36.730 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f3da4001b80 con 0x7f3db009ac40 2024-03-29T10:39:36.730 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.730 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:36.730 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient(hunting): dump: 2024-03-29T10:39:36.730 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.730 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.730 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.730 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.730 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.730 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.730 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.730 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.731 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.731 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.731 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1201062475 0 0) 0x7f3db00b8000 con 0x7f3db00b12b0 2024-03-29T10:39:36.731 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3db0142160 con 0x7f3db00b12b0 2024-03-29T10:39:36.731 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f3d9c0018f0 con 0x7f3db00b12b0 2024-03-29T10:39:36.731 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.731 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:36.731 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient(hunting): dump: 2024-03-29T10:39:36.731 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.731 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.731 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.731 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.731 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.731 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.732 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.732 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.732 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.732 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.732 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2386338457 0 0) 0x7f3db0142f70 con 0x7f3db01d7a40 2024-03-29T10:39:36.732 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 >> v1:172.21.15.57:6790/0 conn(0x7f3db00b12b0 legacy=0x7f3db01d7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.732 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 >> v1:172.21.15.57:6789/0 conn(0x7f3db009ac40 legacy=0x7f3db0142860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.732 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:39:36.732 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 1 monclient: found mon.b 2024-03-29T10:39:36.732 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:36.732 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.732 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db01d7ed0 con 0x7f3db01d7a40 2024-03-29T10:39:36.732 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient: _finish_auth 0 2024-03-29T10:39:36.733 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:39:36.733 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:36.733 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3db6ed9640 5 monclient: authenticate success, global_id 4138 2024-03-29T10:39:36.733 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3db6ed9640 10 monclient: _renew_subs 2024-03-29T10:39:36.733 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3db6ed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.733 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3db6ed9640 1 -- 172.21.15.179:0/1657312323 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3db01e3400 con 0x7f3db01d7a40 2024-03-29T10:39:36.733 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3db6ed9640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:36.733 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3db6ed9640 10 monclient: _renew_subs 2024-03-29T10:39:36.733 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3db6ed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.733 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3db6ed9640 1 -- 172.21.15.179:0/1657312323 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db01e3770 con 0x7f3db01d7a40 2024-03-29T10:39:36.733 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3db6ed9640 1 librados: init done 2024-03-29T10:39:36.733 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3da8003990 con 0x7f3db01d7a40 2024-03-29T10:39:36.733 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:36.733 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f3da8003d20 con 0x7f3db01d7a40 2024-03-29T10:39:36.734 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.734 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:36.734 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 10 monclient: dump: 2024-03-29T10:39:36.734 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.734 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.734 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.734 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.734 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.734 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.734 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.734 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.734 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.734 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.735 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3daeffd640 4 set_mon_vals no callback set 2024-03-29T10:39:36.735 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.728+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3da80042d0 con 0x7f3db01d7a40 2024-03-29T10:39:36.735 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.729+0000 7f3db6ed9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:36.735 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.729+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 1677030536 0 0) 0x7f3da8004580 con 0x7f3db01d7a40 2024-03-29T10:39:36.735 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.729+0000 7f3dad7fa640 10 client.4138.objecter ms_dispatch 0x7f3db0000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T10:39:36.735 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.729+0000 7f3dad7fa640 3 client.4138.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T10:39:36.735 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.729+0000 7f3dad7fa640 3 client.4138.objecter handle_osd_map decoding full epoch 6 2024-03-29T10:39:36.735 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.729+0000 7f3db6ed9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:36.735 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.729+0000 7f3db6ed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.735 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.729+0000 7f3dad7fa640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-29T10:39:36.735 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.729+0000 7f3db6ed9640 1 -- 172.21.15.179:0/1657312323 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3db0143100 con 0x7f3db01d7a40 2024-03-29T10:39:36.735 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.731+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3da8005b40 con 0x7f3db01d7a40 2024-03-29T10:39:36.735 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.731+0000 7f3dad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:36.735 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.731+0000 7f3dad7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:36.743 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:36.759+0000 7f4aa0845040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:39:36.756 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:36.772+0000 7f8972a32040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:39:36.787 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.803+0000 7f3db6ed9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a853ea6e-33dd-42f3-a7a9-0d424e82f27d", "id": 4}] 2024-03-29T10:39:36.787 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.803+0000 7f3db6ed9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a853ea6e-33dd-42f3-a7a9-0d424e82f27d", "id": 4}] 2024-03-29T10:39:36.788 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.803+0000 7f3db6ed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.788 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.803+0000 7f3db6ed9640 1 -- 172.21.15.179:0/1657312323 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a853ea6e-33dd-42f3-a7a9-0d424e82f27d", "id": 4} v 0) v1 -- 0x7f3db013dfe0 con 0x7f3db01d7a40 2024-03-29T10:39:36.794 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.810+0000 7f3dad7fa640 1 -- 172.21.15.179:0/1657312323 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a853ea6e-33dd-42f3-a7a9-0d424e82f27d", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3357956971 0 1697012583) 0x7f3da8005da0 con 0x7f3db01d7a40 2024-03-29T10:39:36.794 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.810+0000 7f3dad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a853ea6e-33dd-42f3-a7a9-0d424e82f27d", "id": 4}] 2024-03-29T10:39:36.794 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.810+0000 7f3dad7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:36.795 INFO:teuthology.orchestra.run.smithi179.stdout:4 2024-03-29T10:39:36.796 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.811+0000 7f3db6ed9640 10 librados: watch_flush enter 2024-03-29T10:39:36.796 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.811+0000 7f3db6ed9640 10 librados: watch_flush exit 2024-03-29T10:39:36.796 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.811+0000 7f3db6ed9640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-29T10:39:36.796 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.811+0000 7f3db6ed9640 10 client.4138.objecter successfully canceled tick 2024-03-29T10:39:36.796 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.811+0000 7f3db6ed9640 10 monclient: shutdown 2024-03-29T10:39:36.796 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.811+0000 7f3db6ed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:36.796 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.811+0000 7f3db6ed9640 1 -- 172.21.15.179:0/1657312323 >> v1:172.21.15.179:6789/0 conn(0x7f3db01d7a40 legacy=0x7f3db01e1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.796 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.811+0000 7f3db6ed9640 1 -- 172.21.15.179:0/1657312323 shutdown_connections 2024-03-29T10:39:36.797 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.811+0000 7f3db6ed9640 1 -- 172.21.15.179:0/1657312323 >> 172.21.15.179:0/1657312323 conn(0x7f3db0066b00 msgr2=0x7f3db013d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:36.797 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.812+0000 7f3db6ed9640 1 -- 172.21.15.179:0/1657312323 shutdown_connections 2024-03-29T10:39:36.797 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.812+0000 7f3db6ed9640 1 -- 172.21.15.179:0/1657312323 wait complete. 2024-03-29T10:39:36.797 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.812+0000 7f3db6ed9640 1 librados: shutdown 2024-03-29T10:39:36.797 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:36.813+0000 7f4aa0845040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:39:36.806 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 2d808cb3-4df1-4247-a61b-7c32fe4954ba 5 2024-03-29T10:39:36.810 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:36.827+0000 7f8972a32040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:39:36.846 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:36.862+0000 7f4aa0845040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:39:36.859 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:36.876+0000 7f8972a32040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:39:36.946 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.962+0000 7f5729663640 10 monclient: get_monmap_and_config 2024-03-29T10:39:36.947 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.962+0000 7f5729663640 10 monclient: build_initial_monmap 2024-03-29T10:39:36.947 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.962+0000 7f5729663640 10 monclient: monmap: 2024-03-29T10:39:36.947 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T10:39:36.947 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.947 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:39:36.963354+0000 2024-03-29T10:39:36.947 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:39:36.963354+0000 2024-03-29T10:39:36.948 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:36.948 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.948 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:36.948 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:39:36.948 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:39:36.948 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.948 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.962+0000 7f5729663640 1 Processor -- start 2024-03-29T10:39:36.948 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.963+0000 7f5729663640 1 -- start start 2024-03-29T10:39:36.948 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.963+0000 7f5729663640 10 monclient: init 2024-03-29T10:39:36.958 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.963+0000 7f5729663640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:36.959 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.963+0000 7f5729663640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:39:36.959 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.963+0000 7f5729663640 10 monclient(hunting): picked mon.noname-a con 0x7f57240b8240 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:36.959 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.963+0000 7f5729663640 10 monclient(hunting): picked mon.noname-b con 0x7f57240b1f60 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:36.959 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.963+0000 7f5729663640 10 monclient(hunting): picked mon.noname-c con 0x7f572409c8c0 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:36.959 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.963+0000 7f5729663640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572409bd80 con 0x7f57240b8240 2024-03-29T10:39:36.959 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.963+0000 7f5729663640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572409b960 con 0x7f572409c8c0 2024-03-29T10:39:36.959 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.963+0000 7f5729663640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57240b2ec0 con 0x7f57240b1f60 2024-03-29T10:39:36.959 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.963+0000 7f5729663640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:36.959 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.963+0000 7f5729663640 10 monclient(hunting): authenticate will time out at 1089.232813s 2024-03-29T10:39:36.960 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.963+0000 7f57237fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f572409c8c0 0x7f572414d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:35172/0 (socket says 172.21.15.179:35172) 2024-03-29T10:39:36.960 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.963+0000 7f57237fe640 1 -- 172.21.15.179:0/2979198571 learned_addr learned my addr 172.21.15.179:0/2979198571 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T10:39:36.960 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 1 -- 172.21.15.179:0/2979198571 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587874555 0 0) 0x7f572409b960 con 0x7f572409c8c0 2024-03-29T10:39:36.960 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.960 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.960 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.960 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 10 monclient(hunting): my global_id is 4141 2024-03-29T10:39:36.960 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 1 -- 172.21.15.179:0/2979198571 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f570c003520 con 0x7f572409c8c0 2024-03-29T10:39:36.961 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 1 -- 172.21.15.179:0/2979198571 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1670994612 0 0) 0x7f572409bd80 con 0x7f57240b8240 2024-03-29T10:39:36.961 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.961 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.961 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.961 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 10 monclient(hunting): my global_id is 4140 2024-03-29T10:39:36.961 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 1 -- 172.21.15.179:0/2979198571 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f572409b960 con 0x7f57240b8240 2024-03-29T10:39:36.961 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 1 -- 172.21.15.179:0/2979198571 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519578344 0 0) 0x7f57240b2ec0 con 0x7f57240b1f60 2024-03-29T10:39:36.961 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.961 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.961 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.961 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 10 monclient(hunting): my global_id is 4145 2024-03-29T10:39:36.962 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 1 -- 172.21.15.179:0/2979198571 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f572409bd80 con 0x7f57240b1f60 2024-03-29T10:39:36.962 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 1 -- 172.21.15.179:0/2979198571 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2510320755 0 0) 0x7f570c003520 con 0x7f572409c8c0 2024-03-29T10:39:36.962 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 1 -- 172.21.15.179:0/2979198571 >> v1:172.21.15.57:6790/0 conn(0x7f57240b1f60 legacy=0x7f572409c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.962 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 1 -- 172.21.15.179:0/2979198571 >> v1:172.21.15.57:6789/0 conn(0x7f57240b8240 legacy=0x7f572413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.962 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:39:36.962 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 1 monclient: found mon.noname-c 2024-03-29T10:39:36.962 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:36.962 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.962 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 1 -- 172.21.15.179:0/2979198571 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572413fb60 con 0x7f572409c8c0 2024-03-29T10:39:36.962 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 10 monclient: _finish_auth 0 2024-03-29T10:39:36.962 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:39:36.962 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:36.962 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5729663640 5 monclient: authenticate success, global_id 4141 2024-03-29T10:39:36.963 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 1 -- 172.21.15.179:0/2979198571 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f571c001a10 con 0x7f572409c8c0 2024-03-29T10:39:36.963 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5729663640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:36.963 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.963 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:36.963 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 10 monclient: dump: 2024-03-29T10:39:36.963 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.963 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.963 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.963 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.963 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.963 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.964 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.964 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.964 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.964 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.964 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5729663640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:36.964 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 1 -- 172.21.15.179:0/2979198571 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f571c001bf0 con 0x7f572409c8c0 2024-03-29T10:39:36.964 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:36.964 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5729663640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:36.964 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 1 -- 172.21.15.179:0/2979198571 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f571c002000 con 0x7f572409c8c0 2024-03-29T10:39:36.964 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.964 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5729663640 4 set_mon_vals no callback set 2024-03-29T10:39:36.964 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:36.964 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5721ffb640 10 monclient: dump: 2024-03-29T10:39:36.964 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.965 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.965 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.965 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.965 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.965 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.965 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.965 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.965 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.965 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.965 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5729663640 10 monclient: shutdown 2024-03-29T10:39:36.965 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5729663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:36.965 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5729663640 1 -- 172.21.15.179:0/2979198571 >> v1:172.21.15.179:6789/0 conn(0x7f572409c8c0 legacy=0x7f572414d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.965 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5729663640 1 -- 172.21.15.179:0/2979198571 shutdown_connections 2024-03-29T10:39:36.966 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.964+0000 7f5729663640 1 -- 172.21.15.179:0/2979198571 >> 172.21.15.179:0/2979198571 conn(0x7f5724066b00 msgr2=0x7f5724066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:36.966 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.965+0000 7f5729663640 1 -- 172.21.15.179:0/2979198571 shutdown_connections 2024-03-29T10:39:36.966 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.965+0000 7f5729663640 1 -- 172.21.15.179:0/2979198571 wait complete. 2024-03-29T10:39:36.966 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.965+0000 7f5729663640 10 monclient: build_initial_monmap 2024-03-29T10:39:36.966 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.965+0000 7f5729663640 10 monclient: monmap: 2024-03-29T10:39:36.966 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T10:39:36.966 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:36.966 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 0.000000 2024-03-29T10:39:36.966 INFO:teuthology.orchestra.run.smithi179.stderr:created 0.000000 2024-03-29T10:39:36.966 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:36.967 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.967 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:36.967 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:39:36.967 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:39:36.967 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.967 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.965+0000 7f5729663640 1 librados: starting msgr at 2024-03-29T10:39:36.967 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.965+0000 7f5729663640 1 librados: starting objecter 2024-03-29T10:39:36.967 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.966+0000 7f5729663640 1 Processor -- start 2024-03-29T10:39:36.967 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.966+0000 7f5729663640 1 -- start start 2024-03-29T10:39:36.967 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.966+0000 7f5729663640 1 librados: setting wanted keys 2024-03-29T10:39:36.967 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.966+0000 7f5729663640 1 librados: calling monclient init 2024-03-29T10:39:36.967 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.966+0000 7f5729663640 10 monclient: init 2024-03-29T10:39:36.967 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.966+0000 7f5729663640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:36.967 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.966+0000 7f5729663640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:39:36.968 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.966+0000 7f5729663640 10 monclient(hunting): picked mon.noname-c con 0x7f572409c8c0 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:36.968 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.966+0000 7f5729663640 10 monclient(hunting): picked mon.noname-b con 0x7f57240b8240 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:36.968 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.966+0000 7f5729663640 10 monclient(hunting): picked mon.noname-a con 0x7f57241324c0 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:36.968 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.966+0000 7f5729663640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5724132aa0 con 0x7f57241324c0 2024-03-29T10:39:36.968 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.966+0000 7f5729663640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5724134030 con 0x7f57240b8240 2024-03-29T10:39:36.968 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.966+0000 7f5729663640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5724134310 con 0x7f572409c8c0 2024-03-29T10:39:36.968 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.966+0000 7f5729663640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:36.968 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.966+0000 7f5729663640 10 monclient(hunting): authenticate will time out at 1089.235879s 2024-03-29T10:39:36.968 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.966+0000 7f57227fc640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f57240b8240 0x7f5724134a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:35180/0 (socket says 172.21.15.179:35180) 2024-03-29T10:39:36.968 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.966+0000 7f57227fc640 1 -- 172.21.15.179:0/1218448619 learned_addr learned my addr 172.21.15.179:0/1218448619 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T10:39:36.969 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.966+0000 7f570b7fe640 10 client.?.objecter ms_handle_connect 0x7f57240b8240 2024-03-29T10:39:36.969 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.966+0000 7f570b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:36.969 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.966+0000 7f570b7fe640 10 client.?.objecter ms_handle_connect 0x7f572409c8c0 2024-03-29T10:39:36.969 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.966+0000 7f570b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:36.969 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.966+0000 7f570b7fe640 10 client.?.objecter ms_handle_connect 0x7f57241324c0 2024-03-29T10:39:36.969 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.966+0000 7f570b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:36.969 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.966+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082234088 0 0) 0x7f5724134030 con 0x7f57240b8240 2024-03-29T10:39:36.969 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.969 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.969 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.969 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient(hunting): my global_id is 4144 2024-03-29T10:39:36.969 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5710004eb0 con 0x7f57240b8240 2024-03-29T10:39:36.969 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436484367 0 0) 0x7f5724134310 con 0x7f572409c8c0 2024-03-29T10:39:36.969 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.969 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.970 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.970 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient(hunting): my global_id is 4148 2024-03-29T10:39:36.970 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5724134030 con 0x7f572409c8c0 2024-03-29T10:39:36.970 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 515468104 0 0) 0x7f5724132aa0 con 0x7f57241324c0 2024-03-29T10:39:36.970 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:36.970 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:36.970 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:36.970 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient(hunting): my global_id is 4143 2024-03-29T10:39:36.970 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5724134310 con 0x7f57241324c0 2024-03-29T10:39:36.970 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3927670506 0 0) 0x7f5710004eb0 con 0x7f57240b8240 2024-03-29T10:39:36.970 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5724132aa0 con 0x7f57240b8240 2024-03-29T10:39:36.970 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f57180019d0 con 0x7f57240b8240 2024-03-29T10:39:36.970 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.970 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:36.971 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient(hunting): dump: 2024-03-29T10:39:36.971 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.971 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.971 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.971 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.971 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.971 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.971 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.971 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.971 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.971 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.971 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:36.971 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2341031889 0 0) 0x7f5724134030 con 0x7f572409c8c0 2024-03-29T10:39:36.971 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5710004eb0 con 0x7f572409c8c0 2024-03-29T10:39:36.971 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f57140017f0 con 0x7f572409c8c0 2024-03-29T10:39:36.972 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.972 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:36.972 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient(hunting): dump: 2024-03-29T10:39:36.972 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.972 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.972 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.972 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.972 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.972 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.972 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.972 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.972 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.972 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.972 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3932427861 0 0) 0x7f5724134310 con 0x7f57241324c0 2024-03-29T10:39:36.972 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5724134030 con 0x7f57241324c0 2024-03-29T10:39:36.973 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f571c001d60 con 0x7f57241324c0 2024-03-29T10:39:36.973 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.973 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:36.973 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient(hunting): dump: 2024-03-29T10:39:36.973 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.973 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.973 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.973 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.973 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.973 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.973 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.973 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.973 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.973 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.974 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3523230811 0 0) 0x7f5724132aa0 con 0x7f57240b8240 2024-03-29T10:39:36.974 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 >> v1:172.21.15.57:6790/0 conn(0x7f572409c8c0 legacy=0x7f5724144e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.974 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 >> v1:172.21.15.57:6789/0 conn(0x7f57241324c0 legacy=0x7f5724132f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:36.974 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:39:36.974 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 1 monclient: found mon.b 2024-03-29T10:39:36.974 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:36.974 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.974 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57241345f0 con 0x7f57240b8240 2024-03-29T10:39:36.974 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:39:36.974 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:39:36.975 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:36.975 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f5729663640 5 monclient: authenticate success, global_id 4144 2024-03-29T10:39:36.975 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f5729663640 10 monclient: _renew_subs 2024-03-29T10:39:36.975 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f5729663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.975 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f5729663640 1 -- 172.21.15.179:0/1218448619 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57241e3330 con 0x7f57240b8240 2024-03-29T10:39:36.975 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f5729663640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:36.975 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f5729663640 10 monclient: _renew_subs 2024-03-29T10:39:36.975 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f5729663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.975 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f5729663640 1 -- 172.21.15.179:0/1218448619 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57241e36a0 con 0x7f57240b8240 2024-03-29T10:39:36.975 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f5729663640 1 librados: init done 2024-03-29T10:39:36.975 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5718001c90 con 0x7f57240b8240 2024-03-29T10:39:36.975 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:36.975 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f5718003a60 con 0x7f57240b8240 2024-03-29T10:39:36.975 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:36.976 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f57217fa640 4 set_mon_vals no callback set 2024-03-29T10:39:36.976 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:36.976 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 monclient: dump: 2024-03-29T10:39:36.976 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:36.976 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:36.976 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.976 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:36.976 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:36.976 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:36.976 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:36.976 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:36.976 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:36.976 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:36.976 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5718003fd0 con 0x7f57240b8240 2024-03-29T10:39:36.977 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 3487337893 0 0) 0x7f5718005d40 con 0x7f57240b8240 2024-03-29T10:39:36.977 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 10 client.4144.objecter ms_dispatch 0x7f5724000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T10:39:36.977 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T10:39:36.977 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2024-03-29T10:39:36.977 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.967+0000 7f570b7fe640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-29T10:39:36.977 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.968+0000 7f5729663640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:36.977 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.968+0000 7f5729663640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:36.977 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.968+0000 7f5729663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:36.977 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.968+0000 7f5729663640 1 -- 172.21.15.179:0/1218448619 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5704005080 con 0x7f57240b8240 2024-03-29T10:39:36.977 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.972+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5718004280 con 0x7f57240b8240 2024-03-29T10:39:36.977 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.972+0000 7f570b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:36.977 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:36.972+0000 7f570b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:37.026 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.042+0000 7f5729663640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2d808cb3-4df1-4247-a61b-7c32fe4954ba", "id": 5}] 2024-03-29T10:39:37.027 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.042+0000 7f5729663640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2d808cb3-4df1-4247-a61b-7c32fe4954ba", "id": 5}] 2024-03-29T10:39:37.027 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.042+0000 7f5729663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:39:37.027 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.042+0000 7f5729663640 1 -- 172.21.15.179:0/1218448619 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd new", "uuid": "2d808cb3-4df1-4247-a61b-7c32fe4954ba", "id": 5} v 0) v1 -- 0x7f57040058b0 con 0x7f57240b8240 2024-03-29T10:39:37.031 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.048+0000 7f570b7fe640 1 -- 172.21.15.179:0/1218448619 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2d808cb3-4df1-4247-a61b-7c32fe4954ba", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 4099897533 0 1988412176) 0x7f57180057c0 con 0x7f57240b8240 2024-03-29T10:39:37.031 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.048+0000 7f570b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2d808cb3-4df1-4247-a61b-7c32fe4954ba", "id": 5}] 2024-03-29T10:39:37.032 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.048+0000 7f570b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:37.032 INFO:teuthology.orchestra.run.smithi179.stdout:5 2024-03-29T10:39:37.034 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.048+0000 7f5729663640 10 librados: watch_flush enter 2024-03-29T10:39:37.034 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.048+0000 7f5729663640 10 librados: watch_flush exit 2024-03-29T10:39:37.034 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.048+0000 7f5729663640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-29T10:39:37.034 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.048+0000 7f5729663640 10 client.4144.objecter successfully canceled tick 2024-03-29T10:39:37.034 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.048+0000 7f5729663640 10 monclient: shutdown 2024-03-29T10:39:37.034 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.048+0000 7f5729663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:37.035 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.048+0000 7f5729663640 1 -- 172.21.15.179:0/1218448619 >> v1:172.21.15.179:6789/0 conn(0x7f57240b8240 legacy=0x7f5724134a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.035 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.049+0000 7f5729663640 1 -- 172.21.15.179:0/1218448619 shutdown_connections 2024-03-29T10:39:37.035 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.049+0000 7f5729663640 1 -- 172.21.15.179:0/1218448619 >> 172.21.15.179:0/1218448619 conn(0x7f5724066b00 msgr2=0x7f572406bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:37.035 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.049+0000 7f5729663640 1 -- 172.21.15.179:0/1218448619 shutdown_connections 2024-03-29T10:39:37.035 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.049+0000 7f5729663640 1 -- 172.21.15.179:0/1218448619 wait complete. 2024-03-29T10:39:37.035 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.049+0000 7f5729663640 1 librados: shutdown 2024-03-29T10:39:37.043 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new 2c2faf76-845d-4a6d-a834-faac22103ef3 6 2024-03-29T10:39:37.143 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.159+0000 7f188de4a640 10 monclient: get_monmap_and_config 2024-03-29T10:39:37.144 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.159+0000 7f188de4a640 10 monclient: build_initial_monmap 2024-03-29T10:39:37.144 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.159+0000 7f188de4a640 10 monclient: monmap: 2024-03-29T10:39:37.144 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T10:39:37.144 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:37.144 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:39:37.160835+0000 2024-03-29T10:39:37.144 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:39:37.160835+0000 2024-03-29T10:39:37.144 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:37.144 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:37.144 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:37.144 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:39:37.144 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:39:37.144 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:37.154 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.160+0000 7f188de4a640 1 Processor -- start 2024-03-29T10:39:37.155 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.160+0000 7f188de4a640 1 -- start start 2024-03-29T10:39:37.155 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.160+0000 7f188de4a640 10 monclient: init 2024-03-29T10:39:37.155 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f188de4a640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:37.155 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f188de4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:39:37.155 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f188de4a640 10 monclient(hunting): picked mon.noname-a con 0x7f188809ac40 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:37.155 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f188de4a640 10 monclient(hunting): picked mon.noname-c con 0x7f188809c870 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:37.156 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f188de4a640 10 monclient(hunting): picked mon.noname-b con 0x7f18880b4360 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:37.156 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f188de4a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f188809dec0 con 0x7f188809ac40 2024-03-29T10:39:37.156 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f188de4a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f188809da30 con 0x7f188809c870 2024-03-29T10:39:37.156 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f188de4a640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18880b39c0 con 0x7f18880b4360 2024-03-29T10:39:37.156 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f188de4a640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:37.156 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f18877fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f188809ac40 0x7f18880b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.179:36812/0 (socket says 172.21.15.179:36812) 2024-03-29T10:39:37.156 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f1887fff640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f18880b4360 0x7f188814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.179:60554/0 (socket says 172.21.15.179:60554) 2024-03-29T10:39:37.157 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f187effd640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f188809c870 0x7f188813f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:35184/0 (socket says 172.21.15.179:35184) 2024-03-29T10:39:37.157 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f18877fe640 1 -- 172.21.15.179:0/1335670713 learned_addr learned my addr 172.21.15.179:0/1335670713 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T10:39:37.157 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f188de4a640 10 monclient(hunting): authenticate will time out at 1089.430714s 2024-03-29T10:39:37.157 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f1886ffd640 1 -- 172.21.15.179:0/1335670713 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 573368841 0 0) 0x7f188809da30 con 0x7f188809c870 2024-03-29T10:39:37.157 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f1886ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:37.157 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f1886ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:37.157 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f1886ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:37.157 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f1886ffd640 10 monclient(hunting): my global_id is 4147 2024-03-29T10:39:37.157 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f1886ffd640 1 -- 172.21.15.179:0/1335670713 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1874003520 con 0x7f188809c870 2024-03-29T10:39:37.158 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f1886ffd640 1 -- 172.21.15.179:0/1335670713 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242203437 0 0) 0x7f1878001040 con 0x7f188809ac40 2024-03-29T10:39:37.158 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f1886ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:37.158 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f1886ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:37.158 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f1886ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:37.158 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f1886ffd640 10 monclient(hunting): my global_id is 4146 2024-03-29T10:39:37.158 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f1886ffd640 1 -- 172.21.15.179:0/1335670713 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f188809da30 con 0x7f188809ac40 2024-03-29T10:39:37.158 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f1886ffd640 1 -- 172.21.15.179:0/1335670713 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 775480122 0 0) 0x7f18880b39c0 con 0x7f18880b4360 2024-03-29T10:39:37.158 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f1886ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:37.158 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f1886ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:37.158 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f1886ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:37.158 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f1886ffd640 10 monclient(hunting): my global_id is 4151 2024-03-29T10:39:37.158 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.161+0000 7f1886ffd640 1 -- 172.21.15.179:0/1335670713 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1878001040 con 0x7f18880b4360 2024-03-29T10:39:37.158 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f1886ffd640 1 -- 172.21.15.179:0/1335670713 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1041869928 0 0) 0x7f1874003520 con 0x7f188809c870 2024-03-29T10:39:37.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f1886ffd640 1 -- 172.21.15.179:0/1335670713 >> v1:172.21.15.57:6790/0 conn(0x7f18880b4360 legacy=0x7f188814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f1886ffd640 1 -- 172.21.15.179:0/1335670713 >> v1:172.21.15.57:6789/0 conn(0x7f188809ac40 legacy=0x7f18880b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f1886ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:39:37.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f1886ffd640 1 monclient: found mon.noname-c 2024-03-29T10:39:37.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f1886ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:37.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f1886ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:39:37.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f1886ffd640 1 -- 172.21.15.179:0/1335670713 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18880b5160 con 0x7f188809c870 2024-03-29T10:39:37.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f1886ffd640 10 monclient: _finish_auth 0 2024-03-29T10:39:37.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f1886ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:39:37.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f1886ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:37.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f1886ffd640 1 -- 172.21.15.179:0/1335670713 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1870001690 con 0x7f188809c870 2024-03-29T10:39:37.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f188de4a640 5 monclient: authenticate success, global_id 4147 2024-03-29T10:39:37.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f188de4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:37.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f1886ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:37.159 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f1886ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:37.160 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f1886ffd640 10 monclient: dump: 2024-03-29T10:39:37.160 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:37.160 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:37.160 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.160 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.160 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:37.160 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:37.160 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:37.160 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:37.160 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:37.160 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:37.160 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f188de4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:37.160 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f1886ffd640 1 -- 172.21.15.179:0/1335670713 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1870001870 con 0x7f188809c870 2024-03-29T10:39:37.160 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f1886ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:37.161 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f1886ffd640 1 -- 172.21.15.179:0/1335670713 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1870001cc0 con 0x7f188809c870 2024-03-29T10:39:37.161 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f188de4a640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:37.161 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f188de4a640 4 set_mon_vals no callback set 2024-03-29T10:39:37.161 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f1886ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:37.161 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f1886ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:37.161 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f1886ffd640 10 monclient: dump: 2024-03-29T10:39:37.161 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:37.161 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:37.161 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.161 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.161 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:37.161 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:37.161 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:37.162 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:37.162 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:37.162 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:37.162 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f188de4a640 10 monclient: shutdown 2024-03-29T10:39:37.162 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f188de4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:37.162 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f188de4a640 1 -- 172.21.15.179:0/1335670713 >> v1:172.21.15.179:6789/0 conn(0x7f188809c870 legacy=0x7f188813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.163 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f188de4a640 1 -- 172.21.15.179:0/1335670713 shutdown_connections 2024-03-29T10:39:37.163 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f188de4a640 1 -- 172.21.15.179:0/1335670713 >> 172.21.15.179:0/1335670713 conn(0x7f1888066b00 msgr2=0x7f1888066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:37.163 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f188de4a640 1 -- 172.21.15.179:0/1335670713 shutdown_connections 2024-03-29T10:39:37.163 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.162+0000 7f188de4a640 1 -- 172.21.15.179:0/1335670713 wait complete. 2024-03-29T10:39:37.163 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.163+0000 7f188de4a640 10 monclient: build_initial_monmap 2024-03-29T10:39:37.163 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.163+0000 7f188de4a640 10 monclient: monmap: 2024-03-29T10:39:37.163 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T10:39:37.163 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:37.163 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 0.000000 2024-03-29T10:39:37.163 INFO:teuthology.orchestra.run.smithi179.stderr:created 0.000000 2024-03-29T10:39:37.163 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:37.163 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:37.163 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:37.164 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:39:37.164 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:39:37.164 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:37.164 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.163+0000 7f188de4a640 1 librados: starting msgr at 2024-03-29T10:39:37.164 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.163+0000 7f188de4a640 1 librados: starting objecter 2024-03-29T10:39:37.164 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.163+0000 7f188de4a640 1 Processor -- start 2024-03-29T10:39:37.164 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f188de4a640 1 -- start start 2024-03-29T10:39:37.164 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f188de4a640 1 librados: setting wanted keys 2024-03-29T10:39:37.164 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f188de4a640 1 librados: calling monclient init 2024-03-29T10:39:37.164 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f188de4a640 10 monclient: init 2024-03-29T10:39:37.164 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f188de4a640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:37.164 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f188de4a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:39:37.164 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f188de4a640 10 monclient(hunting): picked mon.noname-a con 0x7f188809ac40 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:37.164 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f188de4a640 10 monclient(hunting): picked mon.noname-c con 0x7f18880b1180 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:37.164 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f188de4a640 10 monclient(hunting): picked mon.noname-b con 0x7f188813b4d0 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:37.165 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f188de4a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18881cacf0 con 0x7f188809ac40 2024-03-29T10:39:37.165 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f188de4a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18881da930 con 0x7f188813b4d0 2024-03-29T10:39:37.165 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f188de4a640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18881424e0 con 0x7f18880b1180 2024-03-29T10:39:37.165 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f188de4a640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:37.165 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f188de4a640 10 monclient(hunting): authenticate will time out at 1089.433767s 2024-03-29T10:39:37.165 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f1887fff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f188813b4d0 0x7f18881e2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:35200/0 (socket says 172.21.15.179:35200) 2024-03-29T10:39:37.165 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f1887fff640 1 -- 172.21.15.179:0/4051832496 learned_addr learned my addr 172.21.15.179:0/4051832496 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T10:39:37.165 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f1884ff9640 10 client.?.objecter ms_handle_connect 0x7f188813b4d0 2024-03-29T10:39:37.165 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f1884ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:37.165 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f1884ff9640 10 client.?.objecter ms_handle_connect 0x7f188809ac40 2024-03-29T10:39:37.165 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f1884ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:37.165 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f1884ff9640 10 client.?.objecter ms_handle_connect 0x7f18880b1180 2024-03-29T10:39:37.165 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f1884ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:37.165 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343342942 0 0) 0x7f18881da930 con 0x7f188813b4d0 2024-03-29T10:39:37.166 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f1884ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:39:37.166 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f1884ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:37.166 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f1884ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:37.166 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f1884ff9640 10 monclient(hunting): my global_id is 4150 2024-03-29T10:39:37.166 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1868004eb0 con 0x7f188813b4d0 2024-03-29T10:39:37.166 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.164+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367390250 0 0) 0x7f18881cacf0 con 0x7f188809ac40 2024-03-29T10:39:37.166 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:39:37.166 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:37.166 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:37.166 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient(hunting): my global_id is 4149 2024-03-29T10:39:37.166 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18881da930 con 0x7f188809ac40 2024-03-29T10:39:37.166 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82921127 0 0) 0x7f18881424e0 con 0x7f18880b1180 2024-03-29T10:39:37.167 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:39:37.167 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:37.167 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:37.167 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient(hunting): my global_id is 4154 2024-03-29T10:39:37.167 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18881cacf0 con 0x7f18880b1180 2024-03-29T10:39:37.167 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4076014626 0 0) 0x7f18881da930 con 0x7f188809ac40 2024-03-29T10:39:37.167 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18881424e0 con 0x7f188809ac40 2024-03-29T10:39:37.167 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f18780019f0 con 0x7f188809ac40 2024-03-29T10:39:37.167 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:37.167 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:37.167 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient(hunting): dump: 2024-03-29T10:39:37.167 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:37.167 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:37.168 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.168 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.168 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:37.168 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:37.168 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:37.168 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:37.168 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:37.168 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:37.168 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:37.168 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2075564326 0 0) 0x7f1868004eb0 con 0x7f188813b4d0 2024-03-29T10:39:37.168 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18881da930 con 0x7f188813b4d0 2024-03-29T10:39:37.168 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2438466198 0 0) 0x7f18881cacf0 con 0x7f18880b1180 2024-03-29T10:39:37.169 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1868004eb0 con 0x7f18880b1180 2024-03-29T10:39:37.169 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1880001b10 con 0x7f188813b4d0 2024-03-29T10:39:37.169 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:37.169 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:37.169 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient(hunting): dump: 2024-03-29T10:39:37.169 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:37.169 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:37.169 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.169 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.169 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:37.169 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:37.169 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:37.169 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:37.170 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:37.170 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:37.170 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1870001b80 con 0x7f18880b1180 2024-03-29T10:39:37.170 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:37.170 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:37.170 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient(hunting): dump: 2024-03-29T10:39:37.170 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:37.170 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:37.170 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.170 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.170 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:37.170 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:37.170 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:37.170 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:37.170 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:37.171 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:37.171 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 435993302 0 0) 0x7f18881424e0 con 0x7f188809ac40 2024-03-29T10:39:37.171 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 >> v1:172.21.15.57:6790/0 conn(0x7f18880b1180 legacy=0x7f188813adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.171 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 >> v1:172.21.15.179:6789/0 conn(0x7f188813b4d0 legacy=0x7f18881e2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.171 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:39:37.171 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 1 monclient: found mon.a 2024-03-29T10:39:37.171 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:37.171 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:37.171 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1888142670 con 0x7f188809ac40 2024-03-29T10:39:37.171 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient: _finish_auth 0 2024-03-29T10:39:37.171 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:39:37.171 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:37.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f188de4a640 5 monclient: authenticate success, global_id 4149 2024-03-29T10:39:37.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f188de4a640 10 monclient: _renew_subs 2024-03-29T10:39:37.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f188de4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:37.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f188de4a640 1 -- 172.21.15.179:0/4051832496 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18881dae50 con 0x7f188809ac40 2024-03-29T10:39:37.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f188de4a640 10 client.4149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:37.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f188de4a640 10 monclient: _renew_subs 2024-03-29T10:39:37.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f188de4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:37.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f188de4a640 1 -- 172.21.15.179:0/4051832496 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18881da2a0 con 0x7f188809ac40 2024-03-29T10:39:37.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f188de4a640 1 librados: init done 2024-03-29T10:39:37.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1878001cb0 con 0x7f188809ac40 2024-03-29T10:39:37.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:37.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1878004c30 con 0x7f188809ac40 2024-03-29T10:39:37.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:37.172 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:37.173 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f18867fc640 4 set_mon_vals no callback set 2024-03-29T10:39:37.173 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 monclient: dump: 2024-03-29T10:39:37.173 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:37.173 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:37.173 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.173 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.173 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:37.173 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:37.173 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:37.173 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:37.174 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:37.174 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:37.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1878005120 con 0x7f188809ac40 2024-03-29T10:39:37.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 53905032 0 0) 0x7f18780069f0 con 0x7f188809ac40 2024-03-29T10:39:37.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 10 client.4149.objecter ms_dispatch 0x7f1888000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T10:39:37.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 3 client.4149.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T10:39:37.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 3 client.4149.objecter handle_osd_map decoding full epoch 8 2024-03-29T10:39:37.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.165+0000 7f1884ff9640 20 client.4149.objecter dump_active .. 0 homeless 2024-03-29T10:39:37.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.166+0000 7f188de4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:37.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.166+0000 7f188de4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:37.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.166+0000 7f188de4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:37.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.166+0000 7f188de4a640 1 -- 172.21.15.179:0/4051832496 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f186c005080 con 0x7f188809ac40 2024-03-29T10:39:37.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.168+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f18780064b0 con 0x7f188809ac40 2024-03-29T10:39:37.174 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.168+0000 7f1884ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:37.175 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.168+0000 7f1884ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:37.225 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.240+0000 7f188de4a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2c2faf76-845d-4a6d-a834-faac22103ef3", "id": 6}] 2024-03-29T10:39:37.225 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.240+0000 7f188de4a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2c2faf76-845d-4a6d-a834-faac22103ef3", "id": 6}] 2024-03-29T10:39:37.225 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.240+0000 7f188de4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:37.225 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.240+0000 7f188de4a640 1 -- 172.21.15.179:0/4051832496 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd new", "uuid": "2c2faf76-845d-4a6d-a834-faac22103ef3", "id": 6} v 0) v1 -- 0x7f186c0058b0 con 0x7f188809ac40 2024-03-29T10:39:37.226 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.242+0000 7f1884ff9640 1 -- 172.21.15.179:0/4051832496 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2c2faf76-845d-4a6d-a834-faac22103ef3", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 3633935968 0 1113811849) 0x7f1878006720 con 0x7f188809ac40 2024-03-29T10:39:37.226 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.242+0000 7f1884ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2c2faf76-845d-4a6d-a834-faac22103ef3", "id": 6}] 2024-03-29T10:39:37.226 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.242+0000 7f1884ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:37.226 INFO:teuthology.orchestra.run.smithi179.stdout:6 2024-03-29T10:39:37.228 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.243+0000 7f188de4a640 10 librados: watch_flush enter 2024-03-29T10:39:37.228 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.243+0000 7f188de4a640 10 librados: watch_flush exit 2024-03-29T10:39:37.228 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.243+0000 7f188de4a640 20 client.4149.objecter shutdown clearing up homeless session... 2024-03-29T10:39:37.228 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.243+0000 7f188de4a640 10 client.4149.objecter successfully canceled tick 2024-03-29T10:39:37.228 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.243+0000 7f188de4a640 10 monclient: shutdown 2024-03-29T10:39:37.228 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.243+0000 7f188de4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:37.228 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.243+0000 7f188de4a640 1 -- 172.21.15.179:0/4051832496 >> v1:172.21.15.57:6789/0 conn(0x7f188809ac40 legacy=0x7f18881ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.228 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.243+0000 7f188de4a640 1 -- 172.21.15.179:0/4051832496 shutdown_connections 2024-03-29T10:39:37.228 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.243+0000 7f188de4a640 1 -- 172.21.15.179:0/4051832496 >> 172.21.15.179:0/4051832496 conn(0x7f1888066b00 msgr2=0x7f188813bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:37.228 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.243+0000 7f188de4a640 1 -- 172.21.15.179:0/4051832496 shutdown_connections 2024-03-29T10:39:37.228 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.243+0000 7f188de4a640 1 -- 172.21.15.179:0/4051832496 wait complete. 2024-03-29T10:39:37.228 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.243+0000 7f188de4a640 1 librados: shutdown 2024-03-29T10:39:37.237 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --cluster ceph osd new e42d7620-c54f-43e2-96e7-86680f5b7c70 7 2024-03-29T10:39:37.249 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:37.265+0000 7f4aa0845040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:39:37.258 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:37.275+0000 7f8972a32040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:39:37.336 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.352+0000 7fd0444a8640 10 monclient: get_monmap_and_config 2024-03-29T10:39:37.337 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.352+0000 7fd0444a8640 10 monclient: build_initial_monmap 2024-03-29T10:39:37.337 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.352+0000 7fd0444a8640 10 monclient: monmap: 2024-03-29T10:39:37.337 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T10:39:37.337 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:37.337 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:39:37.353511+0000 2024-03-29T10:39:37.337 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:39:37.353511+0000 2024-03-29T10:39:37.337 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:37.337 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:37.337 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:37.337 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:39:37.337 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:39:37.337 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:37.338 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.353+0000 7fd0444a8640 1 Processor -- start 2024-03-29T10:39:37.338 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.353+0000 7fd0444a8640 1 -- start start 2024-03-29T10:39:37.338 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.353+0000 7fd0444a8640 10 monclient: init 2024-03-29T10:39:37.349 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.353+0000 7fd0444a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:37.349 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.353+0000 7fd0444a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:39:37.349 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.353+0000 7fd0444a8640 10 monclient(hunting): picked mon.noname-a con 0x7fd03c09ac40 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:37.349 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.353+0000 7fd0444a8640 10 monclient(hunting): picked mon.noname-c con 0x7fd03c09c870 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:37.349 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.353+0000 7fd0444a8640 10 monclient(hunting): picked mon.noname-b con 0x7fd03c0b4360 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:37.349 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.353+0000 7fd0444a8640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd03c09dec0 con 0x7fd03c09ac40 2024-03-29T10:39:37.349 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.353+0000 7fd0444a8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd03c09da30 con 0x7fd03c09c870 2024-03-29T10:39:37.349 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.353+0000 7fd0444a8640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd03c0b39c0 con 0x7fd03c0b4360 2024-03-29T10:39:37.350 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd0444a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:37.350 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd0444a8640 10 monclient(hunting): authenticate will time out at 1089.623083s 2024-03-29T10:39:37.350 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041a43640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fd03c09c870 0x7fd03c13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:35208/0 (socket says 172.21.15.179:35208) 2024-03-29T10:39:37.350 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041a43640 1 -- 172.21.15.179:0/2576877020 learned_addr learned my addr 172.21.15.179:0/2576877020 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T10:39:37.350 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 1 -- 172.21.15.179:0/2576877020 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946870562 0 0) 0x7fd03c09da30 con 0x7fd03c09c870 2024-03-29T10:39:37.350 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 10 monclient(hunting): _negotiate 2024-03-29T10:39:37.350 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:37.350 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:37.350 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 10 monclient(hunting): my global_id is 4153 2024-03-29T10:39:37.350 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 1 -- 172.21.15.179:0/2576877020 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd028003520 con 0x7fd03c09c870 2024-03-29T10:39:37.350 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 1 -- 172.21.15.179:0/2576877020 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236109560 0 0) 0x7fd03c0b39c0 con 0x7fd03c0b4360 2024-03-29T10:39:37.350 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 10 monclient(hunting): _negotiate 2024-03-29T10:39:37.350 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:37.350 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:37.351 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 10 monclient(hunting): my global_id is 4157 2024-03-29T10:39:37.351 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 1 -- 172.21.15.179:0/2576877020 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd03c09da30 con 0x7fd03c0b4360 2024-03-29T10:39:37.351 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 1 -- 172.21.15.179:0/2576877020 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611145978 0 0) 0x7fd038001110 con 0x7fd03c09ac40 2024-03-29T10:39:37.351 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 10 monclient(hunting): _negotiate 2024-03-29T10:39:37.351 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:37.351 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:37.351 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 10 monclient(hunting): my global_id is 4152 2024-03-29T10:39:37.351 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 1 -- 172.21.15.179:0/2576877020 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd03c0b39c0 con 0x7fd03c09ac40 2024-03-29T10:39:37.351 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 1 -- 172.21.15.179:0/2576877020 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2916373346 0 0) 0x7fd028003520 con 0x7fd03c09c870 2024-03-29T10:39:37.351 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 1 -- 172.21.15.179:0/2576877020 >> v1:172.21.15.57:6790/0 conn(0x7fd03c0b4360 legacy=0x7fd03c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.351 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 1 -- 172.21.15.179:0/2576877020 >> v1:172.21.15.57:6789/0 conn(0x7fd03c09ac40 legacy=0x7fd03c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.351 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 10 monclient: _finish_hunting 0 2024-03-29T10:39:37.351 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 1 monclient: found mon.noname-c 2024-03-29T10:39:37.351 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:37.352 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:39:37.352 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 1 -- 172.21.15.179:0/2576877020 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03c0b5160 con 0x7fd03c09c870 2024-03-29T10:39:37.352 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 10 monclient: _finish_auth 0 2024-03-29T10:39:37.352 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 10 monclient: _check_auth_tickets 2024-03-29T10:39:37.352 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:37.352 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd0444a8640 5 monclient: authenticate success, global_id 4153 2024-03-29T10:39:37.352 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 1 -- 172.21.15.179:0/2576877020 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd02c001690 con 0x7fd03c09c870 2024-03-29T10:39:37.352 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd0444a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:37.352 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:37.352 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:37.352 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 10 monclient: dump: 2024-03-29T10:39:37.352 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:37.352 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:37.352 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.352 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.353 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:37.353 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:37.353 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:37.353 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:37.353 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:37.353 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:37.353 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd0444a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:37.353 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 1 -- 172.21.15.179:0/2576877020 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd02c0017f0 con 0x7fd03c09c870 2024-03-29T10:39:37.353 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:37.353 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 1 -- 172.21.15.179:0/2576877020 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd02c001c00 con 0x7fd03c09c870 2024-03-29T10:39:37.353 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd0444a8640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:37.353 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd0444a8640 4 set_mon_vals no callback set 2024-03-29T10:39:37.353 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:37.354 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:37.354 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.354+0000 7fd041242640 10 monclient: dump: 2024-03-29T10:39:37.354 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:37.354 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:37.354 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.354 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.354 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:37.354 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:37.354 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:37.354 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:37.354 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:37.355 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:37.355 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.355+0000 7fd0444a8640 10 monclient: shutdown 2024-03-29T10:39:37.355 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.355+0000 7fd0444a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:37.355 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.355+0000 7fd0444a8640 1 -- 172.21.15.179:0/2576877020 >> v1:172.21.15.179:6789/0 conn(0x7fd03c09c870 legacy=0x7fd03c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.355 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.355+0000 7fd0444a8640 1 -- 172.21.15.179:0/2576877020 shutdown_connections 2024-03-29T10:39:37.355 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.355+0000 7fd0444a8640 1 -- 172.21.15.179:0/2576877020 >> 172.21.15.179:0/2576877020 conn(0x7fd03c066b00 msgr2=0x7fd03c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:37.355 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.355+0000 7fd0444a8640 1 -- 172.21.15.179:0/2576877020 shutdown_connections 2024-03-29T10:39:37.355 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.355+0000 7fd0444a8640 1 -- 172.21.15.179:0/2576877020 wait complete. 2024-03-29T10:39:37.355 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.355+0000 7fd0444a8640 10 monclient: build_initial_monmap 2024-03-29T10:39:37.355 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.355+0000 7fd0444a8640 10 monclient: monmap: 2024-03-29T10:39:37.355 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 0 2024-03-29T10:39:37.355 INFO:teuthology.orchestra.run.smithi179.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:37.355 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 0.000000 2024-03-29T10:39:37.356 INFO:teuthology.orchestra.run.smithi179.stderr:created 0.000000 2024-03-29T10:39:37.356 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:37.356 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:37.356 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:37.356 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:39:37.356 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:39:37.356 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:37.356 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.356+0000 7fd0444a8640 1 librados: starting msgr at 2024-03-29T10:39:37.356 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.356+0000 7fd0444a8640 1 librados: starting objecter 2024-03-29T10:39:37.357 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.356+0000 7fd0444a8640 1 Processor -- start 2024-03-29T10:39:37.357 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.356+0000 7fd0444a8640 1 -- start start 2024-03-29T10:39:37.357 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.356+0000 7fd0444a8640 1 librados: setting wanted keys 2024-03-29T10:39:37.357 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.356+0000 7fd0444a8640 1 librados: calling monclient init 2024-03-29T10:39:37.357 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.356+0000 7fd0444a8640 10 monclient: init 2024-03-29T10:39:37.357 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.356+0000 7fd0444a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:37.357 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.356+0000 7fd0444a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:37.357 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.356+0000 7fd0444a8640 10 monclient(hunting): picked mon.noname-b con 0x7fd03c09ac40 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:37.358 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.356+0000 7fd0444a8640 10 monclient(hunting): picked mon.noname-c con 0x7fd03c0b07b0 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:37.358 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.356+0000 7fd0444a8640 10 monclient(hunting): picked mon.noname-a con 0x7fd03c1398c0 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:37.358 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.356+0000 7fd0444a8640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd03c0acd70 con 0x7fd03c1398c0 2024-03-29T10:39:37.358 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.356+0000 7fd0444a8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd03c0b0120 con 0x7fd03c09ac40 2024-03-29T10:39:37.358 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd0444a8640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd03c11f8d0 con 0x7fd03c0b07b0 2024-03-29T10:39:37.358 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd0444a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:37.358 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd0444a8640 10 monclient(hunting): authenticate will time out at 1089.626115s 2024-03-29T10:39:37.358 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd042244640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fd03c09ac40 0x7fd03c0afa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.179:35214/0 (socket says 172.21.15.179:35214) 2024-03-29T10:39:37.358 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd042244640 1 -- 172.21.15.179:0/3623365663 learned_addr learned my addr 172.21.15.179:0/3623365663 (peer_addr_for_me v1:172.21.15.179:0/0) 2024-03-29T10:39:37.358 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 client.?.objecter ms_handle_connect 0x7fd03c09ac40 2024-03-29T10:39:37.358 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:37.358 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 client.?.objecter ms_handle_connect 0x7fd03c0b07b0 2024-03-29T10:39:37.358 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:37.358 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 client.?.objecter ms_handle_connect 0x7fd03c1398c0 2024-03-29T10:39:37.359 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:37.359 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024077990 0 0) 0x7fd03c0b0120 con 0x7fd03c09ac40 2024-03-29T10:39:37.359 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:37.359 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:37.359 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:37.359 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient(hunting): my global_id is 4156 2024-03-29T10:39:37.359 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd020004eb0 con 0x7fd03c09ac40 2024-03-29T10:39:37.359 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442411961 0 0) 0x7fd03c11f8d0 con 0x7fd03c0b07b0 2024-03-29T10:39:37.359 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:37.359 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:37.359 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:37.359 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient(hunting): my global_id is 4160 2024-03-29T10:39:37.359 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd03c0b0120 con 0x7fd03c0b07b0 2024-03-29T10:39:37.359 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523773839 0 0) 0x7fd03c0acd70 con 0x7fd03c1398c0 2024-03-29T10:39:37.359 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:37.360 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:37.360 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:37.360 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient(hunting): my global_id is 4155 2024-03-29T10:39:37.360 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd03c11f8d0 con 0x7fd03c1398c0 2024-03-29T10:39:37.360 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2271323237 0 0) 0x7fd03c0b0120 con 0x7fd03c0b07b0 2024-03-29T10:39:37.360 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd03c0acd70 con 0x7fd03c0b07b0 2024-03-29T10:39:37.360 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4066746378 0 0) 0x7fd020004eb0 con 0x7fd03c09ac40 2024-03-29T10:39:37.360 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd03c0b0120 con 0x7fd03c09ac40 2024-03-29T10:39:37.360 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd02c001b00 con 0x7fd03c0b07b0 2024-03-29T10:39:37.360 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:37.360 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:37.361 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient(hunting): dump: 2024-03-29T10:39:37.361 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:37.361 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:37.361 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.361 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.361 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:37.361 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:37.361 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:37.361 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:37.361 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:37.361 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:37.361 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:37.361 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd0380017e0 con 0x7fd03c09ac40 2024-03-29T10:39:37.361 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:37.362 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:37.362 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient(hunting): dump: 2024-03-29T10:39:37.362 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:37.362 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:37.362 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.362 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.362 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:37.362 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:37.362 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:37.362 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:37.362 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:37.362 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:37.362 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4062803345 0 0) 0x7fd03c11f8d0 con 0x7fd03c1398c0 2024-03-29T10:39:37.362 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd020004eb0 con 0x7fd03c1398c0 2024-03-29T10:39:37.363 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd034001bd0 con 0x7fd03c1398c0 2024-03-29T10:39:37.363 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:37.363 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:37.363 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient(hunting): dump: 2024-03-29T10:39:37.363 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:37.363 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:37.363 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.363 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.363 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:37.363 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:37.363 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:37.363 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:37.363 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:37.363 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:37.364 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1460997628 0 0) 0x7fd03c0acd70 con 0x7fd03c0b07b0 2024-03-29T10:39:37.364 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 >> v1:172.21.15.179:6789/0 conn(0x7fd03c09ac40 legacy=0x7fd03c0afa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.364 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 >> v1:172.21.15.57:6789/0 conn(0x7fd03c1398c0 legacy=0x7fd03c1d7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.364 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:39:37.364 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 1 monclient: found mon.c 2024-03-29T10:39:37.364 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:37.364 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:39:37.364 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03c1d7e20 con 0x7fd03c0b07b0 2024-03-29T10:39:37.364 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient: _finish_auth 0 2024-03-29T10:39:37.364 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:39:37.364 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:37.364 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd0444a8640 5 monclient: authenticate success, global_id 4160 2024-03-29T10:39:37.365 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd0444a8640 10 monclient: _renew_subs 2024-03-29T10:39:37.365 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd0444a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:39:37.365 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd0444a8640 1 -- 172.21.15.179:0/3623365663 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd03c1d80f0 con 0x7fd03c0b07b0 2024-03-29T10:39:37.365 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd0444a8640 10 client.4160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:37.365 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd0444a8640 10 monclient: _renew_subs 2024-03-29T10:39:37.365 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd0444a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:39:37.365 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd0444a8640 1 -- 172.21.15.179:0/3623365663 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd03c1e3e40 con 0x7fd03c0b07b0 2024-03-29T10:39:37.365 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd0444a8640 1 librados: init done 2024-03-29T10:39:37.365 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd02c004a90 con 0x7fd03c0b07b0 2024-03-29T10:39:37.366 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:37.366 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd02c004c00 con 0x7fd03c0b07b0 2024-03-29T10:39:37.366 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd032ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:37.366 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.358+0000 7fd032ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:37.366 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.357+0000 7fd040a41640 4 set_mon_vals no callback set 2024-03-29T10:39:37.366 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.358+0000 7fd032ffd640 10 monclient: dump: 2024-03-29T10:39:37.366 INFO:teuthology.orchestra.run.smithi179.stderr:epoch 1 2024-03-29T10:39:37.366 INFO:teuthology.orchestra.run.smithi179.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:37.366 INFO:teuthology.orchestra.run.smithi179.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.366 INFO:teuthology.orchestra.run.smithi179.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.366 INFO:teuthology.orchestra.run.smithi179.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:37.366 INFO:teuthology.orchestra.run.smithi179.stderr:election_strategy: 1 2024-03-29T10:39:37.366 INFO:teuthology.orchestra.run.smithi179.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:37.366 INFO:teuthology.orchestra.run.smithi179.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:37.367 INFO:teuthology.orchestra.run.smithi179.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:37.367 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:39:37.367 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.358+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd02c005170 con 0x7fd03c0b07b0 2024-03-29T10:39:37.367 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.358+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 75029666 0 0) 0x7fd02c005a80 con 0x7fd03c0b07b0 2024-03-29T10:39:37.367 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.358+0000 7fd032ffd640 10 client.4160.objecter ms_dispatch 0x7fd03c000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T10:39:37.367 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.358+0000 7fd032ffd640 3 client.4160.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T10:39:37.367 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.358+0000 7fd032ffd640 3 client.4160.objecter handle_osd_map decoding full epoch 9 2024-03-29T10:39:37.367 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.358+0000 7fd032ffd640 20 client.4160.objecter dump_active .. 0 homeless 2024-03-29T10:39:37.367 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.358+0000 7fd0444a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:37.367 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.358+0000 7fd0444a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:37.367 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.358+0000 7fd0444a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:39:37.368 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.358+0000 7fd0444a8640 1 -- 172.21.15.179:0/3623365663 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd024005080 con 0x7fd03c0b07b0 2024-03-29T10:39:37.368 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.361+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd02c005dd0 con 0x7fd03c0b07b0 2024-03-29T10:39:37.368 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.361+0000 7fd032ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:37.368 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.361+0000 7fd032ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:37.388 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:37.404+0000 7f4aa0845040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:39:37.400 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:37.416+0000 7f8972a32040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:39:37.416 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.432+0000 7fd0444a8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e42d7620-c54f-43e2-96e7-86680f5b7c70", "id": 7}] 2024-03-29T10:39:37.416 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.432+0000 7fd0444a8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e42d7620-c54f-43e2-96e7-86680f5b7c70", "id": 7}] 2024-03-29T10:39:37.416 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.432+0000 7fd0444a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:39:37.416 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.432+0000 7fd0444a8640 1 -- 172.21.15.179:0/3623365663 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd new", "uuid": "e42d7620-c54f-43e2-96e7-86680f5b7c70", "id": 7} v 0) v1 -- 0x7fd0240058b0 con 0x7fd03c0b07b0 2024-03-29T10:39:37.427 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.443+0000 7fd032ffd640 1 -- 172.21.15.179:0/3623365663 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e42d7620-c54f-43e2-96e7-86680f5b7c70", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 611301403 0 1371665406) 0x7fd02c005420 con 0x7fd03c0b07b0 2024-03-29T10:39:37.427 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.443+0000 7fd032ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e42d7620-c54f-43e2-96e7-86680f5b7c70", "id": 7}] 2024-03-29T10:39:37.427 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.443+0000 7fd032ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:37.428 INFO:teuthology.orchestra.run.smithi179.stdout:7 2024-03-29T10:39:37.429 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.444+0000 7fd0444a8640 10 librados: watch_flush enter 2024-03-29T10:39:37.429 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.444+0000 7fd0444a8640 10 librados: watch_flush exit 2024-03-29T10:39:37.429 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.444+0000 7fd0444a8640 20 client.4160.objecter shutdown clearing up homeless session... 2024-03-29T10:39:37.430 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.444+0000 7fd0444a8640 10 client.4160.objecter successfully canceled tick 2024-03-29T10:39:37.430 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.444+0000 7fd0444a8640 10 monclient: shutdown 2024-03-29T10:39:37.430 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.444+0000 7fd0444a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:37.430 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.444+0000 7fd0444a8640 1 -- 172.21.15.179:0/3623365663 >> v1:172.21.15.57:6790/0 conn(0x7fd03c0b07b0 legacy=0x7fd03c1391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.430 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.444+0000 7fd0444a8640 1 -- 172.21.15.179:0/3623365663 shutdown_connections 2024-03-29T10:39:37.430 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.444+0000 7fd0444a8640 1 -- 172.21.15.179:0/3623365663 >> 172.21.15.179:0/3623365663 conn(0x7fd03c066b00 msgr2=0x7fd03c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:37.430 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.444+0000 7fd0444a8640 1 -- 172.21.15.179:0/3623365663 shutdown_connections 2024-03-29T10:39:37.430 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.444+0000 7fd0444a8640 1 -- 172.21.15.179:0/3623365663 wait complete. 2024-03-29T10:39:37.430 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-29T10:39:37.444+0000 7fd0444a8640 1 librados: shutdown 2024-03-29T10:39:37.438 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T10:39:37.439 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T10:39:37.441 INFO:tasks.ceph.osd.0:Started 2024-03-29T10:39:37.441 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T10:39:37.441 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T10:39:37.443 INFO:tasks.ceph.osd.1:Started 2024-03-29T10:39:37.443 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T10:39:37.443 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T10:39:37.445 INFO:tasks.ceph.osd.2:Started 2024-03-29T10:39:37.446 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T10:39:37.446 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T10:39:37.448 INFO:tasks.ceph.osd.3:Started 2024-03-29T10:39:37.449 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T10:39:37.449 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T10:39:37.481 INFO:tasks.ceph.osd.4:Started 2024-03-29T10:39:37.481 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T10:39:37.481 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T10:39:37.484 INFO:tasks.ceph.osd.5:Started 2024-03-29T10:39:37.484 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T10:39:37.484 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T10:39:37.486 INFO:tasks.ceph.osd.6:Started 2024-03-29T10:39:37.486 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T10:39:37.486 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T10:39:37.488 INFO:tasks.ceph.osd.7:Started 2024-03-29T10:39:37.489 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:37.515 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:37.529+0000 7f4aa0845040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:39:37.532 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:37.548+0000 7f8972a32040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:39:37.581 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:37.597+0000 7f4aa0845040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:39:37.585 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:37.602+0000 7f8972a32040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:39:37.602 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.618+0000 7fd42bfd6640 10 monclient: get_monmap_and_config 2024-03-29T10:39:37.602 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.618+0000 7fd42bfd6640 10 monclient: build_initial_monmap 2024-03-29T10:39:37.602 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.618+0000 7fd42bfd6640 10 monclient: monmap: 2024-03-29T10:39:37.602 INFO:teuthology.orchestra.run.smithi057.stderr:epoch 0 2024-03-29T10:39:37.603 INFO:teuthology.orchestra.run.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:37.603 INFO:teuthology.orchestra.run.smithi057.stderr:last_changed 2024-03-29T10:39:37.619631+0000 2024-03-29T10:39:37.603 INFO:teuthology.orchestra.run.smithi057.stderr:created 2024-03-29T10:39:37.619631+0000 2024-03-29T10:39:37.603 INFO:teuthology.orchestra.run.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:37.603 INFO:teuthology.orchestra.run.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:37.603 INFO:teuthology.orchestra.run.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:37.603 INFO:teuthology.orchestra.run.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:39:37.603 INFO:teuthology.orchestra.run.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:39:37.603 INFO:teuthology.orchestra.run.smithi057.stderr: 2024-03-29T10:39:37.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.619+0000 7fd42bfd6640 1 Processor -- start 2024-03-29T10:39:37.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.619+0000 7fd42bfd6640 1 -- start start 2024-03-29T10:39:37.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.619+0000 7fd42bfd6640 10 monclient: init 2024-03-29T10:39:37.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd42bfd6640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:37.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd42bfd6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:39:37.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd42bfd6640 10 monclient(hunting): picked mon.noname-b con 0x7fd4240b8240 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:37.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd42bfd6640 10 monclient(hunting): picked mon.noname-a con 0x7fd4240b1f60 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:37.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd42bfd6640 10 monclient(hunting): picked mon.noname-c con 0x7fd42409c8c0 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:37.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd42bfd6640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd42409bd80 con 0x7fd4240b1f60 2024-03-29T10:39:37.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd42bfd6640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd42409b960 con 0x7fd42409c8c0 2024-03-29T10:39:37.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd42bfd6640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4240b2ec0 con 0x7fd4240b8240 2024-03-29T10:39:37.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd42bfd6640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:37.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd42bfd6640 10 monclient(hunting): authenticate will time out at 929.796365s 2024-03-29T10:39:37.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd429571640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fd4240b1f60 0x7fd42409c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46198/0 (socket says 172.21.15.57:46198) 2024-03-29T10:39:37.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd429571640 1 -- 172.21.15.57:0/3257389493 learned_addr learned my addr 172.21.15.57:0/3257389493 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:39:37.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd428d70640 1 -- 172.21.15.57:0/3257389493 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098598957 0 0) 0x7fd42409bd80 con 0x7fd4240b1f60 2024-03-29T10:39:37.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd428d70640 10 monclient(hunting): _negotiate 2024-03-29T10:39:37.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd428d70640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:37.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd428d70640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:37.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd428d70640 10 monclient(hunting): my global_id is 4182 2024-03-29T10:39:37.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd428d70640 1 -- 172.21.15.57:0/3257389493 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd410003520 con 0x7fd4240b1f60 2024-03-29T10:39:37.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd428d70640 1 -- 172.21.15.57:0/3257389493 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647021098 0 0) 0x7fd4240b2ec0 con 0x7fd4240b8240 2024-03-29T10:39:37.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd428d70640 10 monclient(hunting): _negotiate 2024-03-29T10:39:37.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd428d70640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:37.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd428d70640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:37.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd428d70640 10 monclient(hunting): my global_id is 4187 2024-03-29T10:39:37.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd428d70640 1 -- 172.21.15.57:0/3257389493 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd42409bd80 con 0x7fd4240b8240 2024-03-29T10:39:37.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd428d70640 1 -- 172.21.15.57:0/3257389493 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1185257472 0 0) 0x7fd42409b960 con 0x7fd42409c8c0 2024-03-29T10:39:37.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd428d70640 10 monclient(hunting): _negotiate 2024-03-29T10:39:37.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd428d70640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:37.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd428d70640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:37.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd428d70640 10 monclient(hunting): my global_id is 4183 2024-03-29T10:39:37.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.620+0000 7fd428d70640 1 -- 172.21.15.57:0/3257389493 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4240b2ec0 con 0x7fd42409c8c0 2024-03-29T10:39:37.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd428d70640 1 -- 172.21.15.57:0/3257389493 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3532266571 0 0) 0x7fd410003520 con 0x7fd4240b1f60 2024-03-29T10:39:37.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd428d70640 1 -- 172.21.15.57:0/3257389493 >> v1:172.21.15.57:6790/0 conn(0x7fd4240b8240 legacy=0x7fd42413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd428d70640 1 -- 172.21.15.57:0/3257389493 >> v1:172.21.15.179:6789/0 conn(0x7fd42409c8c0 legacy=0x7fd42414d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd428d70640 10 monclient: _finish_hunting 0 2024-03-29T10:39:37.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd428d70640 1 monclient: found mon.noname-a 2024-03-29T10:39:37.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd428d70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:37.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd428d70640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:39:37.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd428d70640 1 -- 172.21.15.57:0/3257389493 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd42413fb60 con 0x7fd4240b1f60 2024-03-29T10:39:37.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd428d70640 10 monclient: _finish_auth 0 2024-03-29T10:39:37.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd428d70640 10 monclient: _check_auth_tickets 2024-03-29T10:39:37.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd428d70640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:37.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd42bfd6640 5 monclient: authenticate success, global_id 4182 2024-03-29T10:39:37.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd42bfd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:37.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd428d70640 1 -- 172.21.15.57:0/3257389493 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd414001690 con 0x7fd4240b1f60 2024-03-29T10:39:37.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd428d70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:37.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd428d70640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:37.617 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd428d70640 10 monclient: dump: 2024-03-29T10:39:37.617 INFO:teuthology.orchestra.run.smithi057.stderr:epoch 1 2024-03-29T10:39:37.617 INFO:teuthology.orchestra.run.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:37.618 INFO:teuthology.orchestra.run.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.618 INFO:teuthology.orchestra.run.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.618 INFO:teuthology.orchestra.run.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:37.618 INFO:teuthology.orchestra.run.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:37.618 INFO:teuthology.orchestra.run.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:37.618 INFO:teuthology.orchestra.run.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:37.618 INFO:teuthology.orchestra.run.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:37.618 INFO:teuthology.orchestra.run.smithi057.stderr: 2024-03-29T10:39:37.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd42bfd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:37.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd428d70640 1 -- 172.21.15.57:0/3257389493 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd414001870 con 0x7fd4240b1f60 2024-03-29T10:39:37.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd428d70640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:37.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd428d70640 1 -- 172.21.15.57:0/3257389493 <== mon.0 v1:172.21.15.57:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd414001cc0 con 0x7fd4240b1f60 2024-03-29T10:39:37.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd42bfd6640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:37.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd42bfd6640 4 set_mon_vals no callback set 2024-03-29T10:39:37.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd428d70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:37.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd428d70640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:37.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd428d70640 10 monclient: dump: 2024-03-29T10:39:37.619 INFO:teuthology.orchestra.run.smithi057.stderr:epoch 1 2024-03-29T10:39:37.619 INFO:teuthology.orchestra.run.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:37.619 INFO:teuthology.orchestra.run.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.619 INFO:teuthology.orchestra.run.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.619 INFO:teuthology.orchestra.run.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:37.619 INFO:teuthology.orchestra.run.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:37.619 INFO:teuthology.orchestra.run.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:37.620 INFO:teuthology.orchestra.run.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:37.620 INFO:teuthology.orchestra.run.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:37.620 INFO:teuthology.orchestra.run.smithi057.stderr: 2024-03-29T10:39:37.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd42bfd6640 10 monclient: shutdown 2024-03-29T10:39:37.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd42bfd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:37.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd42bfd6640 1 -- 172.21.15.57:0/3257389493 >> v1:172.21.15.57:6789/0 conn(0x7fd4240b1f60 legacy=0x7fd42409c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd42bfd6640 1 -- 172.21.15.57:0/3257389493 shutdown_connections 2024-03-29T10:39:37.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.621+0000 7fd42bfd6640 1 -- 172.21.15.57:0/3257389493 >> 172.21.15.57:0/3257389493 conn(0x7fd424066b00 msgr2=0x7fd424066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:37.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.622+0000 7fd42bfd6640 1 -- 172.21.15.57:0/3257389493 shutdown_connections 2024-03-29T10:39:37.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.622+0000 7fd42bfd6640 1 -- 172.21.15.57:0/3257389493 wait complete. 2024-03-29T10:39:37.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.622+0000 7fd42bfd6640 10 monclient: build_initial_monmap 2024-03-29T10:39:37.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.622+0000 7fd42bfd6640 10 monclient: monmap: 2024-03-29T10:39:37.621 INFO:teuthology.orchestra.run.smithi057.stderr:epoch 0 2024-03-29T10:39:37.621 INFO:teuthology.orchestra.run.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:37.621 INFO:teuthology.orchestra.run.smithi057.stderr:last_changed 0.000000 2024-03-29T10:39:37.621 INFO:teuthology.orchestra.run.smithi057.stderr:created 0.000000 2024-03-29T10:39:37.621 INFO:teuthology.orchestra.run.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:37.621 INFO:teuthology.orchestra.run.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:37.621 INFO:teuthology.orchestra.run.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:37.621 INFO:teuthology.orchestra.run.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:39:37.621 INFO:teuthology.orchestra.run.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:39:37.621 INFO:teuthology.orchestra.run.smithi057.stderr: 2024-03-29T10:39:37.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.622+0000 7fd42bfd6640 1 librados: starting msgr at 2024-03-29T10:39:37.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.622+0000 7fd42bfd6640 1 librados: starting objecter 2024-03-29T10:39:37.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.623+0000 7fd42bfd6640 1 Processor -- start 2024-03-29T10:39:37.621 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.623+0000 7fd42bfd6640 1 -- start start 2024-03-29T10:39:37.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.623+0000 7fd42bfd6640 1 librados: setting wanted keys 2024-03-29T10:39:37.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.623+0000 7fd42bfd6640 1 librados: calling monclient init 2024-03-29T10:39:37.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.623+0000 7fd42bfd6640 10 monclient: init 2024-03-29T10:39:37.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.623+0000 7fd42bfd6640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:37.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.623+0000 7fd42bfd6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:39:37.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.623+0000 7fd42bfd6640 10 monclient(hunting): picked mon.noname-c con 0x7fd42409c8c0 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:37.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.623+0000 7fd42bfd6640 10 monclient(hunting): picked mon.noname-a con 0x7fd4240b8240 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:37.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.623+0000 7fd42bfd6640 10 monclient(hunting): picked mon.noname-b con 0x7fd4241355b0 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:37.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.623+0000 7fd42bfd6640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4241d8010 con 0x7fd4240b8240 2024-03-29T10:39:37.622 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.623+0000 7fd42bfd6640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd424133e80 con 0x7fd4241355b0 2024-03-29T10:39:37.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.623+0000 7fd42bfd6640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4241d7100 con 0x7fd42409c8c0 2024-03-29T10:39:37.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.623+0000 7fd42bfd6640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:37.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.623+0000 7fd42bfd6640 10 monclient(hunting): authenticate will time out at 929.799659s 2024-03-29T10:39:37.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.623+0000 7fd429d72640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7fd42409c8c0 0x7fd4241d7900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:41006/0 (socket says 172.21.15.57:41006) 2024-03-29T10:39:37.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.623+0000 7fd429571640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fd4240b8240 0x7fd424134ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46200/0 (socket says 172.21.15.57:46200) 2024-03-29T10:39:37.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.623+0000 7fd429d72640 1 -- 172.21.15.57:0/71648987 learned_addr learned my addr 172.21.15.57:0/71648987 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:39:37.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.623+0000 7fd41a7fc640 10 client.?.objecter ms_handle_connect 0x7fd42409c8c0 2024-03-29T10:39:37.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.623+0000 7fd41a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:37.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.623+0000 7fd41a7fc640 10 client.?.objecter ms_handle_connect 0x7fd4240b8240 2024-03-29T10:39:37.623 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.623+0000 7fd41a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:37.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.623+0000 7fd41a7fc640 10 client.?.objecter ms_handle_connect 0x7fd4241355b0 2024-03-29T10:39:37.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.623+0000 7fd41a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:37.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689000037 0 0) 0x7fd4241d8010 con 0x7fd4240b8240 2024-03-29T10:39:37.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:37.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:37.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:37.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient(hunting): my global_id is 4185 2024-03-29T10:39:37.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd408004eb0 con 0x7fd4240b8240 2024-03-29T10:39:37.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419055034 0 0) 0x7fd424133e80 con 0x7fd4241355b0 2024-03-29T10:39:37.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:37.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:37.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:37.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient(hunting): my global_id is 4186 2024-03-29T10:39:37.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4241d8010 con 0x7fd4241355b0 2024-03-29T10:39:37.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169517651 0 0) 0x7fd4241d7100 con 0x7fd42409c8c0 2024-03-29T10:39:37.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:37.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:37.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:37.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient(hunting): my global_id is 4190 2024-03-29T10:39:37.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd424133e80 con 0x7fd42409c8c0 2024-03-29T10:39:37.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 848126740 0 0) 0x7fd408004eb0 con 0x7fd4240b8240 2024-03-29T10:39:37.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4241d7100 con 0x7fd4240b8240 2024-03-29T10:39:37.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1110887969 0 0) 0x7fd424133e80 con 0x7fd42409c8c0 2024-03-29T10:39:37.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd408006ae0 con 0x7fd42409c8c0 2024-03-29T10:39:37.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd420001850 con 0x7fd42409c8c0 2024-03-29T10:39:37.626 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:37.626 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:37.626 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient(hunting): dump: 2024-03-29T10:39:37.626 INFO:teuthology.orchestra.run.smithi057.stderr:epoch 1 2024-03-29T10:39:37.626 INFO:teuthology.orchestra.run.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:37.626 INFO:teuthology.orchestra.run.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.626 INFO:teuthology.orchestra.run.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.626 INFO:teuthology.orchestra.run.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:37.626 INFO:teuthology.orchestra.run.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:37.626 INFO:teuthology.orchestra.run.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:37.626 INFO:teuthology.orchestra.run.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:37.626 INFO:teuthology.orchestra.run.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:37.626 INFO:teuthology.orchestra.run.smithi057.stderr: 2024-03-29T10:39:37.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:37.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd414001b80 con 0x7fd4240b8240 2024-03-29T10:39:37.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:37.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:37.627 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient(hunting): dump: 2024-03-29T10:39:37.627 INFO:teuthology.orchestra.run.smithi057.stderr:epoch 1 2024-03-29T10:39:37.627 INFO:teuthology.orchestra.run.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:37.627 INFO:teuthology.orchestra.run.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.627 INFO:teuthology.orchestra.run.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.627 INFO:teuthology.orchestra.run.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:37.627 INFO:teuthology.orchestra.run.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:37.627 INFO:teuthology.orchestra.run.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:37.627 INFO:teuthology.orchestra.run.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:37.628 INFO:teuthology.orchestra.run.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:37.628 INFO:teuthology.orchestra.run.smithi057.stderr: 2024-03-29T10:39:37.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 179923652 0 0) 0x7fd4241d8010 con 0x7fd4241355b0 2024-03-29T10:39:37.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd424133e80 con 0x7fd4241355b0 2024-03-29T10:39:37.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd41c001c90 con 0x7fd4241355b0 2024-03-29T10:39:37.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:37.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:37.628 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient(hunting): dump: 2024-03-29T10:39:37.628 INFO:teuthology.orchestra.run.smithi057.stderr:epoch 1 2024-03-29T10:39:37.628 INFO:teuthology.orchestra.run.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:37.628 INFO:teuthology.orchestra.run.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.629 INFO:teuthology.orchestra.run.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.629 INFO:teuthology.orchestra.run.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:37.629 INFO:teuthology.orchestra.run.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:37.629 INFO:teuthology.orchestra.run.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:37.629 INFO:teuthology.orchestra.run.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:37.629 INFO:teuthology.orchestra.run.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:37.629 INFO:teuthology.orchestra.run.smithi057.stderr: 2024-03-29T10:39:37.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 637057384 0 0) 0x7fd4241d7100 con 0x7fd4240b8240 2024-03-29T10:39:37.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 >> v1:172.21.15.57:6790/0 conn(0x7fd42409c8c0 legacy=0x7fd4241d7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 >> v1:172.21.15.179:6789/0 conn(0x7fd4241355b0 legacy=0x7fd4241e2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:39:37.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 1 monclient: found mon.a 2024-03-29T10:39:37.629 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:37.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:37.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4241d73e0 con 0x7fd4240b8240 2024-03-29T10:39:37.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:39:37.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:39:37.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:37.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd42bfd6640 5 monclient: authenticate success, global_id 4185 2024-03-29T10:39:37.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd42bfd6640 10 monclient: _renew_subs 2024-03-29T10:39:37.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd42bfd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:37.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd42bfd6640 1 -- 172.21.15.57:0/71648987 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd424131d70 con 0x7fd4240b8240 2024-03-29T10:39:37.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd42bfd6640 10 client.4185.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:37.630 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd42bfd6640 10 monclient: _renew_subs 2024-03-29T10:39:37.631 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd42bfd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:37.631 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd42bfd6640 1 -- 172.21.15.57:0/71648987 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4241325a0 con 0x7fd4240b8240 2024-03-29T10:39:37.631 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd42bfd6640 1 librados: init done 2024-03-29T10:39:37.631 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd414004bf0 con 0x7fd4240b8240 2024-03-29T10:39:37.631 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:37.631 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41bfff640 4 set_mon_vals no callback set 2024-03-29T10:39:37.631 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd414004f00 con 0x7fd4240b8240 2024-03-29T10:39:37.631 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:37.631 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:37.632 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 10 monclient: dump: 2024-03-29T10:39:37.632 INFO:teuthology.orchestra.run.smithi057.stderr:epoch 1 2024-03-29T10:39:37.632 INFO:teuthology.orchestra.run.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:37.632 INFO:teuthology.orchestra.run.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.632 INFO:teuthology.orchestra.run.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:37.632 INFO:teuthology.orchestra.run.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:37.632 INFO:teuthology.orchestra.run.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:37.632 INFO:teuthology.orchestra.run.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:37.632 INFO:teuthology.orchestra.run.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:37.632 INFO:teuthology.orchestra.run.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:37.632 INFO:teuthology.orchestra.run.smithi057.stderr: 2024-03-29T10:39:37.632 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.624+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd4140053f0 con 0x7fd4240b8240 2024-03-29T10:39:37.632 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.625+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7fd414005d70 con 0x7fd4240b8240 2024-03-29T10:39:37.632 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.625+0000 7fd41a7fc640 10 client.4185.objecter ms_dispatch 0x7fd424000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:37.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.625+0000 7fd41a7fc640 3 client.4185.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:37.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.625+0000 7fd41a7fc640 3 client.4185.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:37.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.625+0000 7fd41a7fc640 20 client.4185.objecter dump_active .. 0 homeless 2024-03-29T10:39:37.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.625+0000 7fd42bfd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:37.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.625+0000 7fd42bfd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:37.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.625+0000 7fd42bfd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:37.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.625+0000 7fd42bfd6640 1 -- 172.21.15.57:0/71648987 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd42409b170 con 0x7fd4240b8240 2024-03-29T10:39:37.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.627+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd4140060d0 con 0x7fd4240b8240 2024-03-29T10:39:37.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.627+0000 7fd41a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:37.633 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.627+0000 7fd41a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:37.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.689+0000 7fd42bfd6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:37.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.689+0000 7fd42bfd6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:37.673 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.689+0000 7fd42bfd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:37.674 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.689+0000 7fd42bfd6640 1 -- 172.21.15.57:0/71648987 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4241d7620 con 0x7fd4240b8240 2024-03-29T10:39:37.674 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.689+0000 7fd41a7fc640 1 -- 172.21.15.57:0/71648987 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7fd4241d7620 con 0x7fd4240b8240 2024-03-29T10:39:37.674 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.689+0000 7fd41a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:37.674 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.689+0000 7fd41a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:37.674 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-29T10:39:37.674 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:37.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.690+0000 7fd42bfd6640 10 librados: watch_flush enter 2024-03-29T10:39:37.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.690+0000 7fd42bfd6640 10 librados: watch_flush exit 2024-03-29T10:39:37.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.690+0000 7fd42bfd6640 20 client.4185.objecter shutdown clearing up homeless session... 2024-03-29T10:39:37.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.690+0000 7fd42bfd6640 10 client.4185.objecter successfully canceled tick 2024-03-29T10:39:37.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.690+0000 7fd42bfd6640 10 monclient: shutdown 2024-03-29T10:39:37.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.690+0000 7fd42bfd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:37.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.690+0000 7fd42bfd6640 1 -- 172.21.15.57:0/71648987 >> v1:172.21.15.57:6789/0 conn(0x7fd4240b8240 legacy=0x7fd424134ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:37.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.690+0000 7fd42bfd6640 1 -- 172.21.15.57:0/71648987 shutdown_connections 2024-03-29T10:39:37.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.690+0000 7fd42bfd6640 1 -- 172.21.15.57:0/71648987 >> 172.21.15.57:0/71648987 conn(0x7fd424066b00 msgr2=0x7fd42409c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:37.676 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.690+0000 7fd42bfd6640 1 -- 172.21.15.57:0/71648987 shutdown_connections 2024-03-29T10:39:37.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.690+0000 7fd42bfd6640 1 -- 172.21.15.57:0/71648987 wait complete. 2024-03-29T10:39:37.677 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-29T10:39:37.690+0000 7fd42bfd6640 1 librados: shutdown 2024-03-29T10:39:37.685 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T10:39:37.685 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T10:39:37.692 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:37.709+0000 7f4aa0845040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:39:37.705 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:37.721+0000 7f8972a32040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:39:37.786 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:37.802+0000 7f4aa0845040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:39:37.804 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:37.820+0000 7f8972a32040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:39:37.940 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:37.956+0000 7f4aa0845040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:39:37.964 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:37.981+0000 7f8972a32040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:39:37.981 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:37.997+0000 7f4aa0845040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:39:38.008 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:38.025+0000 7f8972a32040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:39:38.047 INFO:tasks.ceph.osd.1.smithi057.stderr:2024-03-29T10:39:38.063+0000 7fd783668640 -1 Falling back to public interface 2024-03-29T10:39:38.060 INFO:tasks.ceph.osd.3.smithi057.stderr:2024-03-29T10:39:38.076+0000 7fece628b640 -1 Falling back to public interface 2024-03-29T10:39:38.066 INFO:tasks.ceph.osd.2.smithi057.stderr:2024-03-29T10:39:38.083+0000 7f9cc1145640 -1 Falling back to public interface 2024-03-29T10:39:38.071 INFO:tasks.ceph.osd.0.smithi057.stderr:2024-03-29T10:39:38.087+0000 7fcd41c18640 -1 Falling back to public interface 2024-03-29T10:39:38.083 INFO:tasks.ceph.osd.6.smithi179.stderr:2024-03-29T10:39:38.099+0000 7fb42c52a640 -1 Falling back to public interface 2024-03-29T10:39:38.084 INFO:tasks.ceph.osd.4.smithi179.stderr:2024-03-29T10:39:38.100+0000 7f8f664c2640 -1 Falling back to public interface 2024-03-29T10:39:38.087 INFO:tasks.ceph.osd.7.smithi179.stderr:2024-03-29T10:39:38.103+0000 7f6f576d3640 -1 Falling back to public interface 2024-03-29T10:39:38.100 INFO:tasks.ceph.osd.5.smithi179.stderr:2024-03-29T10:39:38.117+0000 7ffadd468640 -1 Falling back to public interface 2024-03-29T10:39:38.139 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:38.155+0000 7f4aa0845040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:39:38.170 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:38.186+0000 7f8972a32040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:39:38.491 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:38.582 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.598+0000 7f195ca31640 10 monclient: get_monmap_and_config 2024-03-29T10:39:38.582 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.598+0000 7f195ca31640 10 monclient: build_initial_monmap 2024-03-29T10:39:38.582 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.598+0000 7f195ca31640 10 monclient: monmap: 2024-03-29T10:39:38.583 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:39:38.583 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:38.583 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:39:38.599536+0000 2024-03-29T10:39:38.583 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:39:38.599536+0000 2024-03-29T10:39:38.583 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:38.583 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:38.583 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:38.583 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:39:38.584 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:39:38.584 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:39:38.592 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.599+0000 7f195ca31640 1 Processor -- start 2024-03-29T10:39:38.592 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.599+0000 7f195ca31640 1 -- start start 2024-03-29T10:39:38.592 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.599+0000 7f195ca31640 10 monclient: init 2024-03-29T10:39:38.592 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195ca31640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:38.592 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195ca31640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:39:38.592 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195ca31640 10 monclient(hunting): picked mon.noname-b con 0x7f19581457f0 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:38.592 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195ca31640 10 monclient(hunting): picked mon.noname-a con 0x7f195813f650 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:38.592 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195ca31640 10 monclient(hunting): picked mon.noname-c con 0x7f19580a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:38.592 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195ca31640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19580aaef0 con 0x7f195813f650 2024-03-29T10:39:38.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195ca31640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1958145bd0 con 0x7f19580a9d70 2024-03-29T10:39:38.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195ca31640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1958145eb0 con 0x7f19581457f0 2024-03-29T10:39:38.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195ca31640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:38.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195ca31640 10 monclient(hunting): authenticate will time out at 930.776280s 2024-03-29T10:39:38.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195659c640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f19581457f0 0x7f195813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:41014/0 (socket says 172.21.15.57:41014) 2024-03-29T10:39:38.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195659c640 1 -- 172.21.15.57:0/786490743 learned_addr learned my addr 172.21.15.57:0/786490743 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:39:38.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 1 -- 172.21.15.57:0/786490743 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4051099328 0 0) 0x7f19580aaef0 con 0x7f195813f650 2024-03-29T10:39:38.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 10 monclient(hunting): _negotiate 2024-03-29T10:39:38.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:38.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:38.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 10 monclient(hunting): my global_id is 4188 2024-03-29T10:39:38.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 1 -- 172.21.15.57:0/786490743 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1944003520 con 0x7f195813f650 2024-03-29T10:39:38.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 1 -- 172.21.15.57:0/786490743 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536073169 0 0) 0x7f1958145eb0 con 0x7f19581457f0 2024-03-29T10:39:38.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 10 monclient(hunting): _negotiate 2024-03-29T10:39:38.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:38.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:38.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 10 monclient(hunting): my global_id is 4193 2024-03-29T10:39:38.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 1 -- 172.21.15.57:0/786490743 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19580aaef0 con 0x7f19581457f0 2024-03-29T10:39:38.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 1 -- 172.21.15.57:0/786490743 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 324314139 0 0) 0x7f1958145bd0 con 0x7f19580a9d70 2024-03-29T10:39:38.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 10 monclient(hunting): _negotiate 2024-03-29T10:39:38.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:38.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:38.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 10 monclient(hunting): my global_id is 4189 2024-03-29T10:39:38.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 1 -- 172.21.15.57:0/786490743 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1958145eb0 con 0x7f19580a9d70 2024-03-29T10:39:38.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 1 -- 172.21.15.57:0/786490743 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3590173510 0 0) 0x7f19580aaef0 con 0x7f19581457f0 2024-03-29T10:39:38.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 1 -- 172.21.15.57:0/786490743 >> v1:172.21.15.179:6789/0 conn(0x7f19580a9d70 legacy=0x7f195814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:38.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 1 -- 172.21.15.57:0/786490743 >> v1:172.21.15.57:6789/0 conn(0x7f195813f650 legacy=0x7f19580b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:38.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 10 monclient: _finish_hunting 0 2024-03-29T10:39:38.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 1 monclient: found mon.noname-b 2024-03-29T10:39:38.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:38.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:39:38.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 1 -- 172.21.15.57:0/786490743 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1958146340 con 0x7f19581457f0 2024-03-29T10:39:38.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 10 monclient: _finish_auth 0 2024-03-29T10:39:38.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 10 monclient: _check_auth_tickets 2024-03-29T10:39:38.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:38.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 1 -- 172.21.15.57:0/786490743 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f19480015a0 con 0x7f19581457f0 2024-03-29T10:39:38.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195ca31640 5 monclient: authenticate success, global_id 4193 2024-03-29T10:39:38.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195ca31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:38.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:38.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:38.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.600+0000 7f195559a640 10 monclient: dump: 2024-03-29T10:39:38.595 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:39:38.595 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:38.596 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:38.596 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:38.596 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:38.596 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:38.596 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:38.596 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:38.596 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:38.596 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:39:38.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.601+0000 7f195ca31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:38.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.601+0000 7f195559a640 1 -- 172.21.15.57:0/786490743 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1948001780 con 0x7f19581457f0 2024-03-29T10:39:38.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.601+0000 7f195559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:38.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.601+0000 7f195ca31640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:38.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.601+0000 7f195559a640 1 -- 172.21.15.57:0/786490743 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1948001bd0 con 0x7f19581457f0 2024-03-29T10:39:38.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.601+0000 7f195559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:38.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.601+0000 7f195ca31640 4 set_mon_vals no callback set 2024-03-29T10:39:38.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.601+0000 7f195559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:38.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.601+0000 7f195559a640 10 monclient: dump: 2024-03-29T10:39:38.597 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:39:38.597 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:38.597 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:38.597 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:38.597 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:38.597 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:38.597 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:38.597 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:38.597 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:38.597 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:39:38.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.601+0000 7f195ca31640 10 monclient: shutdown 2024-03-29T10:39:38.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.601+0000 7f195ca31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:38.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.601+0000 7f195ca31640 1 -- 172.21.15.57:0/786490743 >> v1:172.21.15.57:6790/0 conn(0x7f19581457f0 legacy=0x7f195813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:38.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.601+0000 7f195ca31640 1 -- 172.21.15.57:0/786490743 shutdown_connections 2024-03-29T10:39:38.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.601+0000 7f195ca31640 1 -- 172.21.15.57:0/786490743 >> 172.21.15.57:0/786490743 conn(0x7f1958066a00 msgr2=0x7f1958066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:38.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.601+0000 7f195ca31640 1 -- 172.21.15.57:0/786490743 shutdown_connections 2024-03-29T10:39:38.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.601+0000 7f195ca31640 1 -- 172.21.15.57:0/786490743 wait complete. 2024-03-29T10:39:38.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.602+0000 7f195ca31640 10 monclient: build_initial_monmap 2024-03-29T10:39:38.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.602+0000 7f195ca31640 10 monclient: monmap: 2024-03-29T10:39:38.598 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:39:38.598 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:38.598 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:39:38.598 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:39:38.598 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:38.598 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:38.598 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:38.598 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:39:38.599 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:39:38.599 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:39:38.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f195ca31640 1 librados: starting msgr at 2024-03-29T10:39:38.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f195ca31640 1 librados: starting objecter 2024-03-29T10:39:38.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f195ca31640 1 Processor -- start 2024-03-29T10:39:38.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f195ca31640 1 -- start start 2024-03-29T10:39:38.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f195ca31640 1 librados: setting wanted keys 2024-03-29T10:39:38.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f195ca31640 1 librados: calling monclient init 2024-03-29T10:39:38.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f195ca31640 10 monclient: init 2024-03-29T10:39:38.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f195ca31640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:38.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f195ca31640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:38.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f195ca31640 10 monclient(hunting): picked mon.noname-b con 0x7f19580a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:38.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f195ca31640 10 monclient(hunting): picked mon.noname-c con 0x7f195813f650 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:38.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f195ca31640 10 monclient(hunting): picked mon.noname-a con 0x7f19581457f0 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:38.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f195ca31640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19581403c0 con 0x7f19581457f0 2024-03-29T10:39:38.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f195ca31640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1958144cd0 con 0x7f19580a9d70 2024-03-29T10:39:38.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f195ca31640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1958144e60 con 0x7f195813f650 2024-03-29T10:39:38.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f195ca31640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:38.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f195ca31640 10 monclient(hunting): authenticate will time out at 930.779786s 2024-03-29T10:39:38.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f1955d9b640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f195813f650 0x7f19581d7010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:41026/0 (socket says 172.21.15.57:41026) 2024-03-29T10:39:38.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f1955d9b640 1 -- 172.21.15.57:0/1003901747 learned_addr learned my addr 172.21.15.57:0/1003901747 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:39:38.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f194effd640 10 client.?.objecter ms_handle_connect 0x7f195813f650 2024-03-29T10:39:38.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f194effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:38.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f194effd640 10 client.?.objecter ms_handle_connect 0x7f19581457f0 2024-03-29T10:39:38.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f194effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:38.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f194effd640 10 client.?.objecter ms_handle_connect 0x7f19580a9d70 2024-03-29T10:39:38.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.603+0000 7f194effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:38.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1733841589 0 0) 0x7f1958144e60 con 0x7f195813f650 2024-03-29T10:39:38.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:38.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:38.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:38.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient(hunting): my global_id is 4196 2024-03-29T10:39:38.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1938004eb0 con 0x7f195813f650 2024-03-29T10:39:38.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021049536 0 0) 0x7f19581403c0 con 0x7f19581457f0 2024-03-29T10:39:38.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:38.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:38.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:38.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient(hunting): my global_id is 4191 2024-03-29T10:39:38.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1958144e60 con 0x7f19581457f0 2024-03-29T10:39:38.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2324619509 0 0) 0x7f1958144cd0 con 0x7f19580a9d70 2024-03-29T10:39:38.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:38.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:38.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:38.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient(hunting): my global_id is 4192 2024-03-29T10:39:38.602 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19581403c0 con 0x7f19580a9d70 2024-03-29T10:39:38.602 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3600032409 0 0) 0x7f1958144e60 con 0x7f19581457f0 2024-03-29T10:39:38.602 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1958144cd0 con 0x7f19581457f0 2024-03-29T10:39:38.602 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1950001bf0 con 0x7f19581457f0 2024-03-29T10:39:38.602 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:38.602 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:38.602 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient(hunting): dump: 2024-03-29T10:39:38.602 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:39:38.602 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:38.603 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:38.603 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:38.603 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:38.603 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:38.603 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:38.603 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:38.603 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:38.603 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:39:38.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:38.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2353493842 0 0) 0x7f19581403c0 con 0x7f19580a9d70 2024-03-29T10:39:38.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1958144e60 con 0x7f19580a9d70 2024-03-29T10:39:38.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3739056231 0 0) 0x7f1938004eb0 con 0x7f195813f650 2024-03-29T10:39:38.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19581403c0 con 0x7f195813f650 2024-03-29T10:39:38.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1948001a90 con 0x7f19580a9d70 2024-03-29T10:39:38.604 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:38.604 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:38.604 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient(hunting): dump: 2024-03-29T10:39:38.604 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:39:38.604 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:38.604 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:38.604 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:38.604 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:38.604 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:38.604 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:38.604 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:38.604 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:38.604 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:39:38.604 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1940001a70 con 0x7f195813f650 2024-03-29T10:39:38.604 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:38.604 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:38.605 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient(hunting): dump: 2024-03-29T10:39:38.605 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:39:38.605 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:38.605 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:38.605 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:38.605 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:38.605 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:38.605 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:38.605 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:38.605 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:38.605 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:39:38.605 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3239844748 0 0) 0x7f1958144cd0 con 0x7f19581457f0 2024-03-29T10:39:38.605 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 >> v1:172.21.15.57:6790/0 conn(0x7f195813f650 legacy=0x7f19581d7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:38.605 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 >> v1:172.21.15.179:6789/0 conn(0x7f19580a9d70 legacy=0x7f1958134da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:38.605 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient: _finish_hunting 0 2024-03-29T10:39:38.605 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 1 monclient: found mon.a 2024-03-29T10:39:38.605 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:38.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:38.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1958144ff0 con 0x7f19581457f0 2024-03-29T10:39:38.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient: _finish_auth 0 2024-03-29T10:39:38.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient: _check_auth_tickets 2024-03-29T10:39:38.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:38.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f195ca31640 5 monclient: authenticate success, global_id 4191 2024-03-29T10:39:38.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f195ca31640 10 monclient: _renew_subs 2024-03-29T10:39:38.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f195ca31640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:38.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f195ca31640 1 -- 172.21.15.57:0/1003901747 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19581354b0 con 0x7f19581457f0 2024-03-29T10:39:38.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f195ca31640 10 client.4191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:38.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f195ca31640 10 monclient: _renew_subs 2024-03-29T10:39:38.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f195ca31640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:38.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f195ca31640 1 -- 172.21.15.57:0/1003901747 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f195813df70 con 0x7f19581457f0 2024-03-29T10:39:38.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f195ca31640 1 librados: init done 2024-03-29T10:39:38.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1950002080 con 0x7f19581457f0 2024-03-29T10:39:38.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:38.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1950003b30 con 0x7f19581457f0 2024-03-29T10:39:38.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:38.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:38.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f1954d99640 4 set_mon_vals no callback set 2024-03-29T10:39:38.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 monclient: dump: 2024-03-29T10:39:38.607 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:39:38.607 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:38.607 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:38.607 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:38.607 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:38.607 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:38.607 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:38.607 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:38.607 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:38.608 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:39:38.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1950004020 con 0x7f19581457f0 2024-03-29T10:39:38.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f1950005a40 con 0x7f19581457f0 2024-03-29T10:39:38.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 10 client.4191.objecter ms_dispatch 0x7f1958000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:38.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 3 client.4191.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:38.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 3 client.4191.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:38.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.604+0000 7f194effd640 20 client.4191.objecter dump_active .. 0 homeless 2024-03-29T10:39:38.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.605+0000 7f195ca31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:38.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.605+0000 7f195ca31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:38.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.605+0000 7f195ca31640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:38.609 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.605+0000 7f195ca31640 1 -- 172.21.15.57:0/1003901747 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f193c005080 con 0x7f19581457f0 2024-03-29T10:39:38.609 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.607+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1950004ba0 con 0x7f19581457f0 2024-03-29T10:39:38.609 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.607+0000 7f194effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:38.609 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.607+0000 7f194effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:38.654 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.671+0000 7f195ca31640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:38.654 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.671+0000 7f195ca31640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:38.654 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.671+0000 7f195ca31640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:38.655 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.671+0000 7f195ca31640 1 -- 172.21.15.57:0/1003901747 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f193c005330 con 0x7f19581457f0 2024-03-29T10:39:38.655 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.671+0000 7f194effd640 1 -- 172.21.15.57:0/1003901747 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f1950004500 con 0x7f19581457f0 2024-03-29T10:39:38.655 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.671+0000 7f194effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:38.655 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.671+0000 7f194effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:38.655 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:39:38.656 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:38.657 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.672+0000 7f195ca31640 10 librados: watch_flush enter 2024-03-29T10:39:38.657 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.672+0000 7f195ca31640 10 librados: watch_flush exit 2024-03-29T10:39:38.657 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.672+0000 7f195ca31640 20 client.4191.objecter shutdown clearing up homeless session... 2024-03-29T10:39:38.657 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.672+0000 7f195ca31640 10 client.4191.objecter successfully canceled tick 2024-03-29T10:39:38.657 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.672+0000 7f195ca31640 10 monclient: shutdown 2024-03-29T10:39:38.657 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.672+0000 7f195ca31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:38.657 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.672+0000 7f195ca31640 1 -- 172.21.15.57:0/1003901747 >> v1:172.21.15.57:6789/0 conn(0x7f19581457f0 legacy=0x7f19581e28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:38.657 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.672+0000 7f195ca31640 1 -- 172.21.15.57:0/1003901747 shutdown_connections 2024-03-29T10:39:38.657 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.672+0000 7f195ca31640 1 -- 172.21.15.57:0/1003901747 >> 172.21.15.57:0/1003901747 conn(0x7f1958066a00 msgr2=0x7f19580b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:38.657 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.672+0000 7f195ca31640 1 -- 172.21.15.57:0/1003901747 shutdown_connections 2024-03-29T10:39:38.657 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.672+0000 7f195ca31640 1 -- 172.21.15.57:0/1003901747 wait complete. 2024-03-29T10:39:38.657 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:38.672+0000 7f195ca31640 1 librados: shutdown 2024-03-29T10:39:38.664 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:38.960 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:38.976+0000 7f4aa0845040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:39:39.044 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:39.060+0000 7f8972a32040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:39:39.237 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:39.253+0000 7f4aa0845040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:39:39.288 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:39.305+0000 7f4aa0845040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:39:39.330 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:39.346+0000 7f8972a32040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:39:39.383 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:39.400+0000 7f8972a32040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:39:39.403 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:39.419+0000 7f4aa0845040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:39:39.507 INFO:tasks.ceph.mgr.y.smithi057.stderr:2024-03-29T10:39:39.523+0000 7f8972a32040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:39:40.424 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:40.440+0000 7f4a41511640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T10:39:40.424 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-03-29T10:39:40.441+0000 7f4a41511640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T10:39:41.512 INFO:tasks.ceph.osd.1.smithi057.stderr:2024-03-29T10:39:41.529+0000 7fd783668640 -1 osd.1 0 log_to_monitors true 2024-03-29T10:39:41.552 INFO:tasks.ceph.osd.7.smithi179.stderr:2024-03-29T10:39:41.568+0000 7f6f576d3640 -1 osd.7 0 log_to_monitors true 2024-03-29T10:39:41.578 INFO:tasks.ceph.osd.3.smithi057.stderr:2024-03-29T10:39:41.595+0000 7fece628b640 -1 osd.3 0 log_to_monitors true 2024-03-29T10:39:41.587 INFO:tasks.ceph.osd.0.smithi057.stderr:2024-03-29T10:39:41.603+0000 7fcd41c18640 -1 osd.0 0 log_to_monitors true 2024-03-29T10:39:41.609 INFO:tasks.ceph.osd.2.smithi057.stderr:2024-03-29T10:39:41.626+0000 7f9cc1145640 -1 osd.2 0 log_to_monitors true 2024-03-29T10:39:41.624 INFO:tasks.ceph.osd.6.smithi179.stderr:2024-03-29T10:39:41.641+0000 7fb42c52a640 -1 osd.6 0 log_to_monitors true 2024-03-29T10:39:41.626 INFO:tasks.ceph.osd.4.smithi179.stderr:2024-03-29T10:39:41.642+0000 7f8f664c2640 -1 osd.4 0 log_to_monitors true 2024-03-29T10:39:41.632 INFO:tasks.ceph.osd.5.smithi179.stderr:2024-03-29T10:39:41.648+0000 7ffadd468640 -1 osd.5 0 log_to_monitors true 2024-03-29T10:39:45.474 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:45.620 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.636+0000 7f0fa9eae640 10 monclient: get_monmap_and_config 2024-03-29T10:39:45.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.636+0000 7f0fa9eae640 10 monclient: build_initial_monmap 2024-03-29T10:39:45.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.636+0000 7f0fa9eae640 10 monclient: monmap: 2024-03-29T10:39:45.621 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:39:45.621 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:45.621 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:39:45.637907+0000 2024-03-29T10:39:45.621 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:39:45.637907+0000 2024-03-29T10:39:45.621 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:45.621 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:45.621 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:45.621 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:39:45.621 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:39:45.622 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:39:45.622 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.637+0000 7f0fa9eae640 1 Processor -- start 2024-03-29T10:39:45.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.637+0000 7f0fa9eae640 1 -- start start 2024-03-29T10:39:45.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.637+0000 7f0fa9eae640 10 monclient: init 2024-03-29T10:39:45.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.638+0000 7f0fa9eae640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:45.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.638+0000 7f0fa9eae640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:45.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.638+0000 7f0fa9eae640 10 monclient(hunting): picked mon.noname-c con 0x7f0fa40b0770 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:45.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.639+0000 7f0fa37fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f0fa40b0770 0x7f0fa4149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:37238/0 (socket says 172.21.15.57:37238) 2024-03-29T10:39:45.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.639+0000 7f0fa9eae640 10 monclient(hunting): picked mon.noname-b con 0x7f0fa414a700 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:45.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.639+0000 7f0fa9eae640 10 monclient(hunting): picked mon.noname-a con 0x7f0fa40ac5a0 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:45.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.639+0000 7f0fa37fe640 1 -- 172.21.15.57:0/2026305061 learned_addr learned my addr 172.21.15.57:0/2026305061 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:39:45.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.639+0000 7f0fa9eae640 1 -- 172.21.15.57:0/2026305061 --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fa40aba70 con 0x7f0fa40ac5a0 2024-03-29T10:39:45.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.639+0000 7f0fa9eae640 1 -- 172.21.15.57:0/2026305061 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fa40aecb0 con 0x7f0fa40b0770 2024-03-29T10:39:45.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.639+0000 7f0fa9eae640 1 -- 172.21.15.57:0/2026305061 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fa40af150 con 0x7f0fa414a700 2024-03-29T10:39:45.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.639+0000 7f0fa9eae640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:45.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.639+0000 7f0fa9eae640 10 monclient(hunting): authenticate will time out at 937.816790s 2024-03-29T10:39:45.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.640+0000 7f0fa27fc640 1 -- 172.21.15.57:0/2026305061 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3543714047 0 0) 0x7f0fa40aecb0 con 0x7f0fa40b0770 2024-03-29T10:39:45.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.640+0000 7f0fa27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:45.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.640+0000 7f0fa27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:45.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.640+0000 7f0fa27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:45.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.640+0000 7f0fa27fc640 10 monclient(hunting): my global_id is 4234 2024-03-29T10:39:45.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.640+0000 7f0fa27fc640 1 -- 172.21.15.57:0/2026305061 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f8c003520 con 0x7f0fa40b0770 2024-03-29T10:39:45.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.640+0000 7f0fa27fc640 1 -- 172.21.15.57:0/2026305061 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2217645971 0 0) 0x7f0f8c003520 con 0x7f0fa40b0770 2024-03-29T10:39:45.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.640+0000 7f0fa27fc640 1 -- 172.21.15.57:0/2026305061 >> v1:172.21.15.57:6790/0 conn(0x7f0fa414a700 legacy=0x7f0fa40abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:39:45.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.640+0000 7f0fa27fc640 1 -- 172.21.15.57:0/2026305061 >> v1:172.21.15.57:6789/0 conn(0x7f0fa40ac5a0 legacy=0x7f0fa4151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:39:45.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.640+0000 7f0fa27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:39:45.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.640+0000 7f0fa27fc640 1 monclient: found mon.noname-c 2024-03-29T10:39:45.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.640+0000 7f0fa27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:45.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.640+0000 7f0fa27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:39:45.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.640+0000 7f0fa27fc640 1 -- 172.21.15.57:0/2026305061 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa40ac9d0 con 0x7f0fa40b0770 2024-03-29T10:39:45.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.640+0000 7f0fa27fc640 10 monclient: _finish_auth 0 2024-03-29T10:39:45.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.640+0000 7f0fa27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:39:45.628 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.640+0000 7f0fa27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:45.628 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.640+0000 7f0fa27fc640 1 -- 172.21.15.57:0/2026305061 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f0f940016b0 con 0x7f0fa40b0770 2024-03-29T10:39:45.628 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.640+0000 7f0fa27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:45.628 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.640+0000 7f0fa27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:45.628 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.640+0000 7f0fa27fc640 10 monclient: dump: 2024-03-29T10:39:45.628 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:39:45.628 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:45.628 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:45.628 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:45.628 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:45.628 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:45.628 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:45.628 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:45.628 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:45.628 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:39:45.629 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.640+0000 7f0fa9eae640 5 monclient: authenticate success, global_id 4234 2024-03-29T10:39:45.629 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.640+0000 7f0fa9eae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:45.629 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.641+0000 7f0fa27fc640 1 -- 172.21.15.57:0/2026305061 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f94001940 con 0x7f0fa40b0770 2024-03-29T10:39:45.629 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.641+0000 7f0fa27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:45.629 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.641+0000 7f0fa27fc640 1 -- 172.21.15.57:0/2026305061 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f0f94001cd0 con 0x7f0fa40b0770 2024-03-29T10:39:45.629 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.641+0000 7f0fa27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:45.629 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.641+0000 7f0fa27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:45.629 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.641+0000 7f0fa27fc640 10 monclient: dump: 2024-03-29T10:39:45.629 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:39:45.629 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:45.629 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:45.629 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:45.629 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:45.629 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:45.629 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:45.630 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:45.630 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:45.630 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:39:45.630 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.641+0000 7f0fa9eae640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:45.630 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.641+0000 7f0fa9eae640 4 set_mon_vals no callback set 2024-03-29T10:39:45.630 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.642+0000 7f0fa9eae640 10 monclient: shutdown 2024-03-29T10:39:45.630 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.642+0000 7f0fa9eae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:45.630 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.642+0000 7f0fa9eae640 1 -- 172.21.15.57:0/2026305061 >> v1:172.21.15.179:6789/0 conn(0x7f0fa40b0770 legacy=0x7f0fa4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:45.630 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.642+0000 7f0fa9eae640 1 -- 172.21.15.57:0/2026305061 shutdown_connections 2024-03-29T10:39:45.630 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.642+0000 7f0fa9eae640 1 -- 172.21.15.57:0/2026305061 >> 172.21.15.57:0/2026305061 conn(0x7f0fa406b660 msgr2=0x7f0fa406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:45.630 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.642+0000 7f0fa9eae640 1 -- 172.21.15.57:0/2026305061 shutdown_connections 2024-03-29T10:39:45.630 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.642+0000 7f0fa9eae640 1 -- 172.21.15.57:0/2026305061 wait complete. 2024-03-29T10:39:45.640 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.643+0000 7f0fa9eae640 10 monclient: build_initial_monmap 2024-03-29T10:39:45.640 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.643+0000 7f0fa9eae640 10 monclient: monmap: 2024-03-29T10:39:45.640 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:39:45.640 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:45.640 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:39:45.640 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:39:45.640 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:45.640 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:45.640 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:45.641 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:39:45.641 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:39:45.641 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:39:45.641 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.643+0000 7f0fa9eae640 1 librados: starting msgr at 2024-03-29T10:39:45.641 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.643+0000 7f0fa9eae640 1 librados: starting objecter 2024-03-29T10:39:45.641 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.643+0000 7f0fa9eae640 1 Processor -- start 2024-03-29T10:39:45.641 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.644+0000 7f0fa9eae640 1 -- start start 2024-03-29T10:39:45.641 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.644+0000 7f0fa9eae640 1 librados: setting wanted keys 2024-03-29T10:39:45.641 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.644+0000 7f0fa9eae640 1 librados: calling monclient init 2024-03-29T10:39:45.641 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.644+0000 7f0fa9eae640 10 monclient: init 2024-03-29T10:39:45.641 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.644+0000 7f0fa9eae640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:45.641 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.644+0000 7f0fa9eae640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:39:45.641 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.644+0000 7f0fa9eae640 10 monclient(hunting): picked mon.noname-a con 0x7f0fa40ac5a0 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:45.641 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.644+0000 7f0fa9eae640 10 monclient(hunting): picked mon.noname-c con 0x7f0fa40b0770 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:45.641 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.644+0000 7f0fa9eae640 10 monclient(hunting): picked mon.noname-b con 0x7f0fa414a700 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:45.642 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.645+0000 7f0fa9eae640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fa4128010 con 0x7f0fa40ac5a0 2024-03-29T10:39:45.642 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.645+0000 7f0fa9eae640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fa4128280 con 0x7f0fa414a700 2024-03-29T10:39:45.642 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.645+0000 7f0fa9eae640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fa41df7f0 con 0x7f0fa40b0770 2024-03-29T10:39:45.642 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.645+0000 7f0fa9eae640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:45.642 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.645+0000 7f0fa9eae640 10 monclient(hunting): authenticate will time out at 937.822056s 2024-03-29T10:39:45.642 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.645+0000 7f0fa2ffd640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f0fa40b0770 0x7f0fa41df0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:42026/0 (socket says 172.21.15.57:42026) 2024-03-29T10:39:45.642 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.645+0000 7f0fa2ffd640 1 -- 172.21.15.57:0/414666642 learned_addr learned my addr 172.21.15.57:0/414666642 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:39:45.642 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.645+0000 7f0f8bfff640 10 client.?.objecter ms_handle_connect 0x7f0fa40b0770 2024-03-29T10:39:45.642 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.645+0000 7f0f8bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:45.642 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.645+0000 7f0f8bfff640 10 client.?.objecter ms_handle_connect 0x7f0fa40ac5a0 2024-03-29T10:39:45.642 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.645+0000 7f0f8bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:45.642 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.645+0000 7f0f8bfff640 10 client.?.objecter ms_handle_connect 0x7f0fa414a700 2024-03-29T10:39:45.642 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.645+0000 7f0f8bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:45.643 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.645+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373818285 0 0) 0x7f0fa41df7f0 con 0x7f0fa40b0770 2024-03-29T10:39:45.643 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.646+0000 7f0f8bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:45.643 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.646+0000 7f0f8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:45.643 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.646+0000 7f0f8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:45.643 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.646+0000 7f0f8bfff640 10 monclient(hunting): my global_id is 4241 2024-03-29T10:39:45.643 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.646+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f90004eb0 con 0x7f0fa40b0770 2024-03-29T10:39:45.643 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.646+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3671928667 0 0) 0x7f0fa4128280 con 0x7f0fa414a700 2024-03-29T10:39:45.643 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.646+0000 7f0f8bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:45.643 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.646+0000 7f0f8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:45.643 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.646+0000 7f0f8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:45.643 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.646+0000 7f0f8bfff640 10 monclient(hunting): my global_id is 4237 2024-03-29T10:39:45.643 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.646+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fa41df7f0 con 0x7f0fa414a700 2024-03-29T10:39:45.643 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.646+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2583801559 0 0) 0x7f0fa4128010 con 0x7f0fa40ac5a0 2024-03-29T10:39:45.643 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.646+0000 7f0f8bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:45.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.646+0000 7f0f8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:45.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.646+0000 7f0f8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:45.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.646+0000 7f0f8bfff640 10 monclient(hunting): my global_id is 4230 2024-03-29T10:39:45.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.646+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fa4128280 con 0x7f0fa40ac5a0 2024-03-29T10:39:45.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.646+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1853711490 0 0) 0x7f0fa4128280 con 0x7f0fa40ac5a0 2024-03-29T10:39:45.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.647+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0fa4128010 con 0x7f0fa40ac5a0 2024-03-29T10:39:45.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.647+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 173975324 0 0) 0x7f0f90004eb0 con 0x7f0fa40b0770 2024-03-29T10:39:45.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.647+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0fa4128280 con 0x7f0fa40b0770 2024-03-29T10:39:45.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.647+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 884884104 0 0) 0x7f0fa41df7f0 con 0x7f0fa414a700 2024-03-29T10:39:45.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.647+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f90004eb0 con 0x7f0fa414a700 2024-03-29T10:39:45.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.647+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f0f940013d0 con 0x7f0fa40ac5a0 2024-03-29T10:39:45.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0f8bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:45.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0f8bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:45.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0f8bfff640 10 monclient(hunting): dump: 2024-03-29T10:39:45.644 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:39:45.645 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:45.645 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:45.645 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:45.645 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:45.645 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:45.645 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:45.645 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:45.645 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:45.645 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:39:45.645 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0f8bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:45.645 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f0f98001750 con 0x7f0fa40b0770 2024-03-29T10:39:45.645 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0f8bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:45.645 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0f8bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:45.645 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0f8bfff640 10 monclient(hunting): dump: 2024-03-29T10:39:45.646 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:39:45.646 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:45.646 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:45.646 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:45.646 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:45.646 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:45.646 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:45.646 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:45.646 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:45.646 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:39:45.646 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f0f9c001810 con 0x7f0fa414a700 2024-03-29T10:39:45.646 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0f8bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:45.647 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0f8bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:45.647 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0f8bfff640 10 monclient(hunting): dump: 2024-03-29T10:39:45.647 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:39:45.647 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:45.647 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:45.647 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:45.647 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:45.647 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:45.647 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:45.647 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:45.647 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:45.647 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:39:45.647 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3828778901 0 0) 0x7f0fa4128010 con 0x7f0fa40ac5a0 2024-03-29T10:39:45.647 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 >> v1:172.21.15.57:6790/0 conn(0x7f0fa40b0770 legacy=0x7f0fa41df0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:45.648 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 >> v1:172.21.15.179:6789/0 conn(0x7f0fa414a700 legacy=0x7f0fa4144f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:45.648 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0f8bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:39:45.648 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0f8bfff640 1 monclient: found mon.a 2024-03-29T10:39:45.648 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0f8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:45.648 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0f8bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:45.648 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa41df980 con 0x7f0fa40ac5a0 2024-03-29T10:39:45.648 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0f8bfff640 10 monclient: _finish_auth 0 2024-03-29T10:39:45.648 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0f8bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:39:45.648 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0f8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:45.648 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0fa9eae640 5 monclient: authenticate success, global_id 4230 2024-03-29T10:39:45.648 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0fa9eae640 10 monclient: _renew_subs 2024-03-29T10:39:45.648 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0fa9eae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:45.648 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0fa9eae640 1 -- 172.21.15.57:0/414666642 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fa41239d0 con 0x7f0fa40ac5a0 2024-03-29T10:39:45.648 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0fa9eae640 10 client.4230.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:45.648 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0fa9eae640 10 monclient: _renew_subs 2024-03-29T10:39:45.649 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0fa9eae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:45.649 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0fa9eae640 1 -- 172.21.15.57:0/414666642 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fa4126070 con 0x7f0fa40ac5a0 2024-03-29T10:39:45.649 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.648+0000 7f0fa9eae640 1 librados: init done 2024-03-29T10:39:45.649 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.649+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f94001530 con 0x7f0fa40ac5a0 2024-03-29T10:39:45.649 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.649+0000 7f0f8bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:45.649 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.649+0000 7f0fa1ffb640 4 set_mon_vals no callback set 2024-03-29T10:39:45.649 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.649+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f0f94004c30 con 0x7f0fa40ac5a0 2024-03-29T10:39:45.649 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.649+0000 7f0f8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:45.649 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.649+0000 7f0f8bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:45.649 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.649+0000 7f0f8bfff640 10 monclient: dump: 2024-03-29T10:39:45.649 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:39:45.649 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:45.649 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:45.649 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:45.649 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:45.650 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:45.650 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:45.650 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:45.650 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:45.650 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:39:45.650 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.652+0000 7f0fa9eae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:45.650 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.652+0000 7f0fa9eae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:45.650 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.652+0000 7f0fa9eae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:45.650 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.652+0000 7f0fa9eae640 1 -- 172.21.15.57:0/414666642 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f84005080 con 0x7f0fa40ac5a0 2024-03-29T10:39:45.650 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.653+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f0f9401ac00 con 0x7f0fa40ac5a0 2024-03-29T10:39:45.650 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.654+0000 7f0fa2ffd640 10 monclient: get_auth_request con 0x7f0f9006e2d0 auth_method 0 2024-03-29T10:39:45.650 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.654+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f0f940849d0 con 0x7f0fa40ac5a0 2024-03-29T10:39:45.650 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.654+0000 7f0f8bfff640 10 client.4230.objecter ms_dispatch 0x7f0fa4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:45.650 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.654+0000 7f0f8bfff640 3 client.4230.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:45.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.654+0000 7f0f8bfff640 3 client.4230.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:45.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.654+0000 7f0f8bfff640 20 client.4230.objecter dump_active .. 0 homeless 2024-03-29T10:39:45.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.654+0000 7f0f8bfff640 10 client.4230.objecter ms_handle_connect 0x7f0f9006e2d0 2024-03-29T10:39:45.657 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.673+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f940502b0 con 0x7f0fa40ac5a0 2024-03-29T10:39:45.657 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.673+0000 7f0f8bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:45.658 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:45.673+0000 7f0f8bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:46.025 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:46.040+0000 7f0fa9eae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:46.025 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:46.041+0000 7f0fa9eae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:46.025 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:46.041+0000 7f0fa9eae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:46.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:46.041+0000 7f0fa9eae640 1 -- 172.21.15.57:0/414666642 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f84005600 con 0x7f0fa40ac5a0 2024-03-29T10:39:46.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:46.042+0000 7f0f8bfff640 1 -- 172.21.15.57:0/414666642 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f0f940521e0 con 0x7f0fa40ac5a0 2024-03-29T10:39:46.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:46.042+0000 7f0f8bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:46.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:46.042+0000 7f0f8bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:46.027 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:39:46.027 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:46.031 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:46.045+0000 7f0fa9eae640 10 librados: watch_flush enter 2024-03-29T10:39:46.032 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:46.045+0000 7f0fa9eae640 10 librados: watch_flush exit 2024-03-29T10:39:46.032 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:46.045+0000 7f0fa9eae640 20 client.4230.objecter shutdown clearing up homeless session... 2024-03-29T10:39:46.032 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:46.045+0000 7f0fa9eae640 10 client.4230.objecter successfully canceled tick 2024-03-29T10:39:46.032 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:46.045+0000 7f0fa9eae640 1 -- 172.21.15.57:0/414666642 >> v1:172.21.15.179:6816/2698295936 conn(0x7f0f9006e2d0 legacy=0x7f0f90070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:46.032 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:46.046+0000 7f0fa9eae640 10 monclient: shutdown 2024-03-29T10:39:46.033 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:46.046+0000 7f0fa9eae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:46.033 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:46.046+0000 7f0fa9eae640 1 -- 172.21.15.57:0/414666642 >> v1:172.21.15.57:6789/0 conn(0x7f0fa40ac5a0 legacy=0x7f0fa4126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:46.033 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:46.046+0000 7f0fa9eae640 1 -- 172.21.15.57:0/414666642 shutdown_connections 2024-03-29T10:39:46.033 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:46.046+0000 7f0fa9eae640 1 -- 172.21.15.57:0/414666642 >> 172.21.15.57:0/414666642 conn(0x7f0fa406b660 msgr2=0x7f0fa406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:46.033 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:46.046+0000 7f0fa9eae640 1 -- 172.21.15.57:0/414666642 shutdown_connections 2024-03-29T10:39:46.033 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:46.047+0000 7f0fa9eae640 1 -- 172.21.15.57:0/414666642 wait complete. 2024-03-29T10:39:46.033 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:46.047+0000 7f0fa9eae640 1 librados: shutdown 2024-03-29T10:39:46.046 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:52.853 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:52.991 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.007+0000 7ff1cf5d6640 10 monclient: get_monmap_and_config 2024-03-29T10:39:52.992 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.007+0000 7ff1cf5d6640 10 monclient: build_initial_monmap 2024-03-29T10:39:52.992 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.007+0000 7ff1cf5d6640 10 monclient: monmap: 2024-03-29T10:39:52.992 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:39:52.992 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:52.992 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:39:53.008884+0000 2024-03-29T10:39:52.992 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:39:53.008884+0000 2024-03-29T10:39:52.992 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:52.992 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:52.992 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:52.992 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:39:52.992 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:39:52.992 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:39:52.999 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.008+0000 7ff1cf5d6640 1 Processor -- start 2024-03-29T10:39:52.999 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.008+0000 7ff1cf5d6640 1 -- start start 2024-03-29T10:39:52.999 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.008+0000 7ff1cf5d6640 10 monclient: init 2024-03-29T10:39:53.000 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.009+0000 7ff1cf5d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:53.000 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.009+0000 7ff1cf5d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:53.000 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.009+0000 7ff1cf5d6640 10 monclient(hunting): picked mon.noname-c con 0x7ff1c81457f0 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:53.000 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.009+0000 7ff1cf5d6640 10 monclient(hunting): picked mon.noname-b con 0x7ff1c813f650 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:53.000 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.009+0000 7ff1cf5d6640 10 monclient(hunting): picked mon.noname-a con 0x7ff1c80a9d70 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:53.000 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.009+0000 7ff1cf5d6640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c80aaef0 con 0x7ff1c80a9d70 2024-03-29T10:39:53.000 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.009+0000 7ff1cf5d6640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c8145bd0 con 0x7ff1c81457f0 2024-03-29T10:39:53.000 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.009+0000 7ff1cf5d6640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c8145eb0 con 0x7ff1c813f650 2024-03-29T10:39:53.000 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.009+0000 7ff1cf5d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:53.000 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.009+0000 7ff1cf5d6640 10 monclient(hunting): authenticate will time out at 945.186420s 2024-03-29T10:39:53.000 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.009+0000 7ff1cdb73640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ff1c80a9d70 0x7ff1c814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:57724/0 (socket says 172.21.15.57:57724) 2024-03-29T10:39:53.000 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.009+0000 7ff1cdb73640 1 -- 172.21.15.57:0/1956168731 learned_addr learned my addr 172.21.15.57:0/1956168731 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:39:53.000 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 1 -- 172.21.15.57:0/1956168731 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584326964 0 0) 0x7ff1c8145eb0 con 0x7ff1c813f650 2024-03-29T10:39:53.001 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:53.001 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:53.001 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:53.001 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 10 monclient(hunting): my global_id is 4244 2024-03-29T10:39:53.001 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 1 -- 172.21.15.57:0/1956168731 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1b0003520 con 0x7ff1c813f650 2024-03-29T10:39:53.001 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 1 -- 172.21.15.57:0/1956168731 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1488634238 0 0) 0x7ff1c8145bd0 con 0x7ff1c81457f0 2024-03-29T10:39:53.001 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:53.001 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:53.001 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:53.001 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 10 monclient(hunting): my global_id is 4240 2024-03-29T10:39:53.001 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 1 -- 172.21.15.57:0/1956168731 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1b0003840 con 0x7ff1c81457f0 2024-03-29T10:39:53.001 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 1 -- 172.21.15.57:0/1956168731 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795968596 0 0) 0x7ff1c80aaef0 con 0x7ff1c80a9d70 2024-03-29T10:39:53.001 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:53.002 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:53.002 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:53.002 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 10 monclient(hunting): my global_id is 4233 2024-03-29T10:39:53.002 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 1 -- 172.21.15.57:0/1956168731 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1c8145bd0 con 0x7ff1c80a9d70 2024-03-29T10:39:53.002 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 1 -- 172.21.15.57:0/1956168731 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2670500633 0 0) 0x7ff1b0003520 con 0x7ff1c813f650 2024-03-29T10:39:53.002 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 1 -- 172.21.15.57:0/1956168731 >> v1:172.21.15.179:6789/0 conn(0x7ff1c81457f0 legacy=0x7ff1c813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:53.002 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 1 -- 172.21.15.57:0/1956168731 >> v1:172.21.15.57:6789/0 conn(0x7ff1c80a9d70 legacy=0x7ff1c814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:53.002 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:39:53.002 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 1 monclient: found mon.noname-b 2024-03-29T10:39:53.002 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:53.002 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:39:53.002 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 1 -- 172.21.15.57:0/1956168731 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c8146340 con 0x7ff1c813f650 2024-03-29T10:39:53.002 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 10 monclient: _finish_auth 0 2024-03-29T10:39:53.002 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:39:53.003 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:53.003 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.010+0000 7ff1cf5d6640 5 monclient: authenticate success, global_id 4244 2024-03-29T10:39:53.003 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.011+0000 7ff1cf5d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:53.003 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.011+0000 7ff1c7fff640 1 -- 172.21.15.57:0/1956168731 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7ff1bc001670 con 0x7ff1c813f650 2024-03-29T10:39:53.003 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.011+0000 7ff1c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:53.003 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.011+0000 7ff1c7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:53.003 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.011+0000 7ff1c7fff640 10 monclient: dump: 2024-03-29T10:39:53.003 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:39:53.003 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:53.003 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:53.004 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:53.004 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:53.004 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:53.004 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:53.004 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:53.004 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:53.004 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:39:53.004 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.011+0000 7ff1cf5d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:53.004 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.011+0000 7ff1c7fff640 1 -- 172.21.15.57:0/1956168731 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1bc001850 con 0x7ff1c813f650 2024-03-29T10:39:53.004 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.011+0000 7ff1c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:53.004 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.011+0000 7ff1cf5d6640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:53.004 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.011+0000 7ff1cf5d6640 4 set_mon_vals no callback set 2024-03-29T10:39:53.005 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.011+0000 7ff1c7fff640 1 -- 172.21.15.57:0/1956168731 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7ff1bc001670 con 0x7ff1c813f650 2024-03-29T10:39:53.005 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.011+0000 7ff1c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:53.005 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.011+0000 7ff1c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:53.005 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.011+0000 7ff1c7fff640 10 monclient: dump: 2024-03-29T10:39:53.005 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:39:53.005 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:53.005 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:53.005 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:53.005 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:53.005 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:53.005 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:53.005 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:53.005 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:53.005 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:39:53.005 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.012+0000 7ff1cf5d6640 10 monclient: shutdown 2024-03-29T10:39:53.006 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.012+0000 7ff1cf5d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:53.006 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.012+0000 7ff1cf5d6640 1 -- 172.21.15.57:0/1956168731 >> v1:172.21.15.57:6790/0 conn(0x7ff1c813f650 legacy=0x7ff1c80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:53.006 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.012+0000 7ff1cf5d6640 1 -- 172.21.15.57:0/1956168731 shutdown_connections 2024-03-29T10:39:53.006 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.012+0000 7ff1cf5d6640 1 -- 172.21.15.57:0/1956168731 >> 172.21.15.57:0/1956168731 conn(0x7ff1c8066a00 msgr2=0x7ff1c8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:53.006 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.012+0000 7ff1cf5d6640 1 -- 172.21.15.57:0/1956168731 shutdown_connections 2024-03-29T10:39:53.006 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.012+0000 7ff1cf5d6640 1 -- 172.21.15.57:0/1956168731 wait complete. 2024-03-29T10:39:53.006 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.013+0000 7ff1cf5d6640 10 monclient: build_initial_monmap 2024-03-29T10:39:53.006 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.013+0000 7ff1cf5d6640 10 monclient: monmap: 2024-03-29T10:39:53.006 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:39:53.006 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:53.006 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:39:53.006 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:39:53.006 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:53.006 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:53.007 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:39:53.007 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:39:53.007 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:39:53.007 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:39:53.007 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.014+0000 7ff1cf5d6640 1 librados: starting msgr at 2024-03-29T10:39:53.007 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.014+0000 7ff1cf5d6640 1 librados: starting objecter 2024-03-29T10:39:53.007 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.014+0000 7ff1cf5d6640 1 Processor -- start 2024-03-29T10:39:53.007 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.014+0000 7ff1cf5d6640 1 -- start start 2024-03-29T10:39:53.007 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.014+0000 7ff1cf5d6640 1 librados: setting wanted keys 2024-03-29T10:39:53.007 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.014+0000 7ff1cf5d6640 1 librados: calling monclient init 2024-03-29T10:39:53.007 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.014+0000 7ff1cf5d6640 10 monclient: init 2024-03-29T10:39:53.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.016+0000 7ff1cf5d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:53.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.016+0000 7ff1cf5d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:39:53.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.016+0000 7ff1cf5d6640 10 monclient(hunting): picked mon.noname-c con 0x7ff1c80a9d70 addr v1:172.21.15.57:6790/0 2024-03-29T10:39:53.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.016+0000 7ff1cf5d6640 10 monclient(hunting): picked mon.noname-a con 0x7ff1c813f650 addr v1:172.21.15.57:6789/0 2024-03-29T10:39:53.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.016+0000 7ff1cf5d6640 10 monclient(hunting): picked mon.noname-b con 0x7ff1c81457f0 addr v1:172.21.15.179:6789/0 2024-03-29T10:39:53.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.016+0000 7ff1cf5d6640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c806e860 con 0x7ff1c813f650 2024-03-29T10:39:53.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.016+0000 7ff1ccb71640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ff1c813f650 0x7ff1c8142430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:58604/0 (socket says 172.21.15.57:58604) 2024-03-29T10:39:53.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.016+0000 7ff1ccb71640 1 -- 172.21.15.57:0/2074183656 learned_addr learned my addr 172.21.15.57:0/2074183656 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:39:53.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.016+0000 7ff1cf5d6640 1 -- 172.21.15.57:0/2074183656 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c8141fa0 con 0x7ff1c81457f0 2024-03-29T10:39:53.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.016+0000 7ff1c5ffb640 10 client.?.objecter ms_handle_connect 0x7ff1c813f650 2024-03-29T10:39:53.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.016+0000 7ff1c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:53.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.016+0000 7ff1c5ffb640 1 -- 172.21.15.57:0/2074183656 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913981520 0 0) 0x7ff1c806e860 con 0x7ff1c813f650 2024-03-29T10:39:53.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.017+0000 7ff1cf5d6640 1 -- 172.21.15.57:0/2074183656 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c813a740 con 0x7ff1c80a9d70 2024-03-29T10:39:53.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.017+0000 7ff1cf5d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:53.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.017+0000 7ff1cf5d6640 10 monclient(hunting): authenticate will time out at 945.194267s 2024-03-29T10:39:53.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.017+0000 7ff1c5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:53.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.017+0000 7ff1c5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:53.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.017+0000 7ff1c5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:53.013 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.017+0000 7ff1c5ffb640 10 monclient(hunting): my global_id is 4236 2024-03-29T10:39:53.013 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.017+0000 7ff1c5ffb640 1 -- 172.21.15.57:0/2074183656 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1b4004eb0 con 0x7ff1c813f650 2024-03-29T10:39:53.013 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.017+0000 7ff1c5ffb640 1 -- 172.21.15.57:0/2074183656 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1518060214 0 0) 0x7ff1b4004eb0 con 0x7ff1c813f650 2024-03-29T10:39:53.013 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.017+0000 7ff1c5ffb640 1 -- 172.21.15.57:0/2074183656 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1c806e860 con 0x7ff1c813f650 2024-03-29T10:39:53.013 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.017+0000 7ff1c5ffb640 10 client.?.objecter ms_handle_connect 0x7ff1c80a9d70 2024-03-29T10:39:53.013 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.017+0000 7ff1c5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:53.013 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.017+0000 7ff1c5ffb640 1 -- 172.21.15.57:0/2074183656 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7ff1bc001d80 con 0x7ff1c813f650 2024-03-29T10:39:53.013 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.017+0000 7ff1c5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:53.013 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.017+0000 7ff1c5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:53.013 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.017+0000 7ff1c5ffb640 10 monclient(hunting): dump: 2024-03-29T10:39:53.014 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:39:53.014 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:53.014 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:53.014 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:53.014 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:53.014 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:53.014 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:53.014 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:53.014 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:53.014 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:39:53.014 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.017+0000 7ff1c5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:53.014 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1c5ffb640 1 -- 172.21.15.57:0/2074183656 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1995111040 0 0) 0x7ff1c806e860 con 0x7ff1c813f650 2024-03-29T10:39:53.014 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1c5ffb640 1 -- 172.21.15.57:0/2074183656 >> v1:172.21.15.57:6790/0 conn(0x7ff1c80a9d70 legacy=0x7ff1c8139660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:53.014 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1c5ffb640 1 -- 172.21.15.57:0/2074183656 >> v1:172.21.15.179:6789/0 conn(0x7ff1c81457f0 legacy=0x7ff1c81dda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:53.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1c5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:39:53.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1c5ffb640 1 monclient: found mon.a 2024-03-29T10:39:53.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1c5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:53.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1c5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:53.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1c5ffb640 1 -- 172.21.15.57:0/2074183656 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c8142b40 con 0x7ff1c813f650 2024-03-29T10:39:53.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1c5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:39:53.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1c5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:39:53.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:53.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1c5ffb640 1 -- 172.21.15.57:0/2074183656 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1bc003a80 con 0x7ff1c813f650 2024-03-29T10:39:53.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:53.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1c5ffb640 1 -- 172.21.15.57:0/2074183656 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7ff1bc003e90 con 0x7ff1c813f650 2024-03-29T10:39:53.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:53.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1c77fe640 4 set_mon_vals no callback set 2024-03-29T10:39:53.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1c5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:53.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1c5ffb640 10 monclient: dump: 2024-03-29T10:39:53.016 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:39:53.016 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:39:53.016 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:53.016 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:39:53.016 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:53.016 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:39:53.016 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:39:53.016 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:39:53.016 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:39:53.016 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:39:53.016 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1cf5d6640 5 monclient: authenticate success, global_id 4236 2024-03-29T10:39:53.016 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1cf5d6640 10 monclient: _renew_subs 2024-03-29T10:39:53.016 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1cf5d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:53.017 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1cf5d6640 1 -- 172.21.15.57:0/2074183656 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c8141870 con 0x7ff1c813f650 2024-03-29T10:39:53.017 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1cf5d6640 10 client.4236.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:53.017 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1cf5d6640 10 monclient: _renew_subs 2024-03-29T10:39:53.017 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1cf5d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:53.017 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1cf5d6640 1 -- 172.21.15.57:0/2074183656 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1c813ff80 con 0x7ff1c813f650 2024-03-29T10:39:53.017 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.018+0000 7ff1cf5d6640 1 librados: init done 2024-03-29T10:39:53.017 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.020+0000 7ff1cf5d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:53.017 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.020+0000 7ff1cf5d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:53.017 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.020+0000 7ff1cf5d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:53.017 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.020+0000 7ff1cf5d6640 1 -- 172.21.15.57:0/2074183656 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c8141c10 con 0x7ff1c813f650 2024-03-29T10:39:53.018 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.022+0000 7ff1c5ffb640 1 -- 172.21.15.57:0/2074183656 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7ff1bc003ff0 con 0x7ff1c813f650 2024-03-29T10:39:53.018 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.023+0000 7ff1c5ffb640 1 -- 172.21.15.57:0/2074183656 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7ff1bc084cb0 con 0x7ff1c813f650 2024-03-29T10:39:53.018 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.023+0000 7ff1c5ffb640 10 client.4236.objecter ms_dispatch 0x7ff1c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:53.018 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.023+0000 7ff1c5ffb640 3 client.4236.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:53.018 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.023+0000 7ff1c5ffb640 3 client.4236.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:53.018 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.023+0000 7ff1c5ffb640 20 client.4236.objecter dump_active .. 0 homeless 2024-03-29T10:39:53.018 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.023+0000 7ff1cd372640 10 monclient: get_auth_request con 0x7ff1b406dbb0 auth_method 0 2024-03-29T10:39:53.018 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.023+0000 7ff1c5ffb640 10 client.4236.objecter ms_handle_connect 0x7ff1b406dbb0 2024-03-29T10:39:53.018 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.029+0000 7ff1c5ffb640 1 -- 172.21.15.57:0/2074183656 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff1bc050590 con 0x7ff1c813f650 2024-03-29T10:39:53.019 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.029+0000 7ff1c5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:53.019 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.029+0000 7ff1c5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:53.429 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.444+0000 7ff1cf5d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:53.429 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.444+0000 7ff1cf5d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:53.429 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.444+0000 7ff1cf5d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:39:53.429 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.444+0000 7ff1cf5d6640 1 -- 172.21.15.57:0/2074183656 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1c8141a00 con 0x7ff1c813f650 2024-03-29T10:39:53.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.446+0000 7ff1c5ffb640 1 -- 172.21.15.57:0/2074183656 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7ff1bc0524c0 con 0x7ff1c813f650 2024-03-29T10:39:53.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.446+0000 7ff1c5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:53.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.446+0000 7ff1c5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:53.431 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:39:53.431 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:53.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.449+0000 7ff1cf5d6640 10 librados: watch_flush enter 2024-03-29T10:39:53.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.449+0000 7ff1cf5d6640 10 librados: watch_flush exit 2024-03-29T10:39:53.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.449+0000 7ff1cf5d6640 20 client.4236.objecter shutdown clearing up homeless session... 2024-03-29T10:39:53.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.449+0000 7ff1cf5d6640 10 client.4236.objecter successfully canceled tick 2024-03-29T10:39:53.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.449+0000 7ff1cf5d6640 1 -- 172.21.15.57:0/2074183656 >> v1:172.21.15.179:6816/2698295936 conn(0x7ff1b406dbb0 legacy=0x7ff1b4070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:53.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.449+0000 7ff1cf5d6640 10 monclient: shutdown 2024-03-29T10:39:53.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.449+0000 7ff1cf5d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:53.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.449+0000 7ff1cf5d6640 1 -- 172.21.15.57:0/2074183656 >> v1:172.21.15.57:6789/0 conn(0x7ff1c813f650 legacy=0x7ff1c8142430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:53.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.450+0000 7ff1cf5d6640 1 -- 172.21.15.57:0/2074183656 shutdown_connections 2024-03-29T10:39:53.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.450+0000 7ff1cf5d6640 1 -- 172.21.15.57:0/2074183656 >> 172.21.15.57:0/2074183656 conn(0x7ff1c8066a00 msgr2=0x7ff1c813b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:53.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.450+0000 7ff1cf5d6640 1 -- 172.21.15.57:0/2074183656 shutdown_connections 2024-03-29T10:39:53.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.450+0000 7ff1cf5d6640 1 -- 172.21.15.57:0/2074183656 wait complete. 2024-03-29T10:39:53.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:39:53.450+0000 7ff1cf5d6640 1 librados: shutdown 2024-03-29T10:39:53.451 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:00.258 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:00.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.431+0000 7f3737036640 10 monclient: get_monmap_and_config 2024-03-29T10:40:00.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.431+0000 7f3737036640 10 monclient: build_initial_monmap 2024-03-29T10:40:00.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.431+0000 7f3737036640 10 monclient: monmap: 2024-03-29T10:40:00.416 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:40:00.416 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:00.416 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:40:00.433216+0000 2024-03-29T10:40:00.416 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:40:00.433216+0000 2024-03-29T10:40:00.416 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:00.417 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:00.417 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:40:00.417 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:40:00.417 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:40:00.417 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:00.419 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.432+0000 7f3737036640 1 Processor -- start 2024-03-29T10:40:00.419 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.432+0000 7f3737036640 1 -- start start 2024-03-29T10:40:00.419 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.432+0000 7f3737036640 10 monclient: init 2024-03-29T10:40:00.419 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.433+0000 7f3737036640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:00.419 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.433+0000 7f3737036640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:00.419 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.433+0000 7f3737036640 10 monclient(hunting): picked mon.noname-c con 0x7f3728037210 addr v1:172.21.15.179:6789/0 2024-03-29T10:40:00.419 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.433+0000 7f3737036640 10 monclient(hunting): picked mon.noname-b con 0x7f372802b0d0 addr v1:172.21.15.57:6790/0 2024-03-29T10:40:00.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.433+0000 7f3737036640 10 monclient(hunting): picked mon.noname-a con 0x7f37280c8240 addr v1:172.21.15.57:6789/0 2024-03-29T10:40:00.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.433+0000 7f3737036640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37280308a0 con 0x7f37280c8240 2024-03-29T10:40:00.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.433+0000 7f3737036640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37280c8690 con 0x7f3728037210 2024-03-29T10:40:00.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.433+0000 7f3737036640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37280c1bd0 con 0x7f372802b0d0 2024-03-29T10:40:00.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.433+0000 7f3737036640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:00.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.433+0000 7f3737036640 10 monclient(hunting): authenticate will time out at 952.610803s 2024-03-29T10:40:00.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.435+0000 7f3736034640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f3728037210 0x7f37280c14c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:54562/0 (socket says 172.21.15.57:54562) 2024-03-29T10:40:00.421 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.435+0000 7f3736034640 1 -- 172.21.15.57:0/3541359299 learned_addr learned my addr 172.21.15.57:0/3541359299 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:40:00.421 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.437+0000 7f3735032640 1 -- 172.21.15.57:0/3541359299 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228648725 0 0) 0x7f37280308a0 con 0x7f37280c8240 2024-03-29T10:40:00.421 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.437+0000 7f3735032640 10 monclient(hunting): _negotiate 2024-03-29T10:40:00.422 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.437+0000 7f3735032640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:00.422 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.437+0000 7f3735032640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:00.422 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.437+0000 7f3735032640 10 monclient(hunting): my global_id is 4239 2024-03-29T10:40:00.422 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.437+0000 7f3735032640 1 -- 172.21.15.57:0/3541359299 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f371c003520 con 0x7f37280c8240 2024-03-29T10:40:00.424 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.437+0000 7f3735032640 1 -- 172.21.15.57:0/3541359299 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4235100705 0 0) 0x7f371c003520 con 0x7f37280c8240 2024-03-29T10:40:00.424 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.437+0000 7f3735032640 1 -- 172.21.15.57:0/3541359299 >> v1:172.21.15.57:6790/0 conn(0x7f372802b0d0 legacy=0x7f37280c7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:00.424 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.437+0000 7f3735032640 1 -- 172.21.15.57:0/3541359299 >> v1:172.21.15.179:6789/0 conn(0x7f3728037210 legacy=0x7f37280c14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:00.424 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.437+0000 7f3735032640 10 monclient: _finish_hunting 0 2024-03-29T10:40:00.424 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.437+0000 7f3735032640 1 monclient: found mon.noname-a 2024-03-29T10:40:00.424 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.437+0000 7f3735032640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:00.424 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.437+0000 7f3735032640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:40:00.424 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.437+0000 7f3735032640 1 -- 172.21.15.57:0/3541359299 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37280c5390 con 0x7f37280c8240 2024-03-29T10:40:00.424 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.437+0000 7f3735032640 10 monclient: _finish_auth 0 2024-03-29T10:40:00.424 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.437+0000 7f3735032640 10 monclient: _check_auth_tickets 2024-03-29T10:40:00.425 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.437+0000 7f3735032640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:00.425 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.437+0000 7f3735032640 1 -- 172.21.15.57:0/3541359299 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f373004f730 con 0x7f37280c8240 2024-03-29T10:40:00.425 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.437+0000 7f3735032640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:00.425 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.437+0000 7f3735032640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:00.425 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.437+0000 7f3735032640 10 monclient: dump: 2024-03-29T10:40:00.426 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:00.426 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:00.426 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:00.426 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:00.426 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:00.426 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:00.426 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:00.426 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:00.426 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:00.426 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:00.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.438+0000 7f3735032640 1 -- 172.21.15.57:0/3541359299 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37300008d0 con 0x7f37280c8240 2024-03-29T10:40:00.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.438+0000 7f3735032640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:00.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.438+0000 7f3735032640 1 -- 172.21.15.57:0/3541359299 <== mon.0 v1:172.21.15.57:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f37300758b0 con 0x7f37280c8240 2024-03-29T10:40:00.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.438+0000 7f3735032640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:00.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.438+0000 7f3735032640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:00.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.438+0000 7f3735032640 10 monclient: dump: 2024-03-29T10:40:00.427 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:00.427 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:00.427 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:00.427 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:00.427 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:00.427 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:00.427 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:00.427 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:00.428 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:00.428 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:00.428 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.438+0000 7f3737036640 5 monclient: authenticate success, global_id 4239 2024-03-29T10:40:00.428 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.438+0000 7f3737036640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:00.428 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.438+0000 7f3737036640 4 set_mon_vals no callback set 2024-03-29T10:40:00.428 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.439+0000 7f3737036640 10 monclient: shutdown 2024-03-29T10:40:00.428 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.439+0000 7f3737036640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:00.428 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.439+0000 7f3737036640 1 -- 172.21.15.57:0/3541359299 >> v1:172.21.15.57:6789/0 conn(0x7f37280c8240 legacy=0x7f37280d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:00.428 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.439+0000 7f3737036640 1 -- 172.21.15.57:0/3541359299 shutdown_connections 2024-03-29T10:40:00.428 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.439+0000 7f3737036640 1 -- 172.21.15.57:0/3541359299 >> 172.21.15.57:0/3541359299 conn(0x7f372803a550 msgr2=0x7f37280367e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:00.428 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.439+0000 7f3737036640 1 -- 172.21.15.57:0/3541359299 shutdown_connections 2024-03-29T10:40:00.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.439+0000 7f3737036640 1 -- 172.21.15.57:0/3541359299 wait complete. 2024-03-29T10:40:00.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.440+0000 7f3737036640 10 monclient: build_initial_monmap 2024-03-29T10:40:00.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.440+0000 7f3737036640 10 monclient: monmap: 2024-03-29T10:40:00.431 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:40:00.431 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:00.431 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:40:00.431 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:40:00.431 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:00.431 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:00.431 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:40:00.432 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:40:00.432 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:40:00.432 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:00.432 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.441+0000 7f3737036640 1 librados: starting msgr at 2024-03-29T10:40:00.432 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.441+0000 7f3737036640 1 librados: starting objecter 2024-03-29T10:40:00.432 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.441+0000 7f3737036640 1 Processor -- start 2024-03-29T10:40:00.432 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.441+0000 7f3737036640 1 -- start start 2024-03-29T10:40:00.432 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.441+0000 7f3737036640 1 librados: setting wanted keys 2024-03-29T10:40:00.432 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.441+0000 7f3737036640 1 librados: calling monclient init 2024-03-29T10:40:00.432 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.441+0000 7f3737036640 10 monclient: init 2024-03-29T10:40:00.432 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.442+0000 7f3737036640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:00.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.442+0000 7f3737036640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:00.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.442+0000 7f3737036640 10 monclient(hunting): picked mon.noname-b con 0x7f3728037210 addr v1:172.21.15.179:6789/0 2024-03-29T10:40:00.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.442+0000 7f3737036640 10 monclient(hunting): picked mon.noname-c con 0x7f37280c8240 addr v1:172.21.15.57:6790/0 2024-03-29T10:40:00.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.442+0000 7f3736034640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f3728037210 0x7f37280b6eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:54656/0 (socket says 172.21.15.57:54656) 2024-03-29T10:40:00.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.442+0000 7f3737036640 10 monclient(hunting): picked mon.noname-a con 0x7f3728033c90 addr v1:172.21.15.57:6789/0 2024-03-29T10:40:00.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.442+0000 7f3737036640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37280cb070 con 0x7f3728033c90 2024-03-29T10:40:00.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.442+0000 7f3737036640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3728030150 con 0x7f3728037210 2024-03-29T10:40:00.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.442+0000 7f3737036640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3728030ce0 con 0x7f37280c8240 2024-03-29T10:40:00.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.442+0000 7f3737036640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:00.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.442+0000 7f3737036640 10 monclient(hunting): authenticate will time out at 952.619909s 2024-03-29T10:40:00.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.442+0000 7f3736034640 1 -- 172.21.15.57:0/3613689929 learned_addr learned my addr 172.21.15.57:0/3613689929 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:40:00.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.443+0000 7f3726ffd640 10 client.?.objecter ms_handle_connect 0x7f3728037210 2024-03-29T10:40:00.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.443+0000 7f3726ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:00.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.446+0000 7f3726ffd640 10 client.?.objecter ms_handle_connect 0x7f3728033c90 2024-03-29T10:40:00.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.446+0000 7f3726ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:00.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.446+0000 7f3726ffd640 10 client.?.objecter ms_handle_connect 0x7f37280c8240 2024-03-29T10:40:00.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.446+0000 7f3726ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:00.440 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.448+0000 7f3726ffd640 1 -- 172.21.15.57:0/3613689929 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1515754796 0 0) 0x7f3728030150 con 0x7f3728037210 2024-03-29T10:40:00.440 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.448+0000 7f3726ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:00.440 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.448+0000 7f3726ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:00.440 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.448+0000 7f3726ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:00.440 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.448+0000 7f3726ffd640 10 monclient(hunting): my global_id is 4243 2024-03-29T10:40:00.440 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.448+0000 7f3726ffd640 1 -- 172.21.15.57:0/3613689929 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3714004eb0 con 0x7f3728037210 2024-03-29T10:40:00.440 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.448+0000 7f3726ffd640 1 -- 172.21.15.57:0/3613689929 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362531570 0 0) 0x7f37280cb070 con 0x7f3728033c90 2024-03-29T10:40:00.440 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.448+0000 7f3726ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:00.440 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.448+0000 7f3726ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:00.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.448+0000 7f3726ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:00.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.448+0000 7f3726ffd640 10 monclient(hunting): my global_id is 4242 2024-03-29T10:40:00.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.448+0000 7f3726ffd640 1 -- 172.21.15.57:0/3613689929 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3728030150 con 0x7f3728033c90 2024-03-29T10:40:00.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.448+0000 7f3726ffd640 1 -- 172.21.15.57:0/3613689929 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 154643468 0 0) 0x7f3714004eb0 con 0x7f3728037210 2024-03-29T10:40:00.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.448+0000 7f3726ffd640 1 -- 172.21.15.57:0/3613689929 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37280cb070 con 0x7f3728037210 2024-03-29T10:40:00.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.448+0000 7f3726ffd640 1 -- 172.21.15.57:0/3613689929 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f3720001930 con 0x7f3728037210 2024-03-29T10:40:00.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.449+0000 7f3726ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:00.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.449+0000 7f3726ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:00.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.449+0000 7f3726ffd640 10 monclient(hunting): dump: 2024-03-29T10:40:00.441 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:00.441 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:00.441 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:00.442 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:00.442 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:00.442 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:00.442 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:00.442 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:00.442 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:00.442 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:00.442 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.449+0000 7f3726ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:00.442 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.449+0000 7f3726ffd640 1 -- 172.21.15.57:0/3613689929 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 858805706 0 0) 0x7f3728030150 con 0x7f3728033c90 2024-03-29T10:40:00.442 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.449+0000 7f3726ffd640 1 -- 172.21.15.57:0/3613689929 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3714004eb0 con 0x7f3728033c90 2024-03-29T10:40:00.442 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.449+0000 7f3726ffd640 1 -- 172.21.15.57:0/3613689929 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f373004f400 con 0x7f3728033c90 2024-03-29T10:40:00.443 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.449+0000 7f3726ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:00.443 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.449+0000 7f3726ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:00.443 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.449+0000 7f3726ffd640 10 monclient(hunting): dump: 2024-03-29T10:40:00.443 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:00.443 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:00.443 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:00.443 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:00.443 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:00.443 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:00.443 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:00.443 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:00.444 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:00.444 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:00.444 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.449+0000 7f3726ffd640 1 -- 172.21.15.57:0/3613689929 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4277718727 0 0) 0x7f3714004eb0 con 0x7f3728033c90 2024-03-29T10:40:00.444 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.449+0000 7f3726ffd640 1 -- 172.21.15.57:0/3613689929 >> v1:172.21.15.57:6790/0 conn(0x7f37280c8240 legacy=0x7f37280c6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:00.444 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.449+0000 7f3726ffd640 1 -- 172.21.15.57:0/3613689929 >> v1:172.21.15.179:6789/0 conn(0x7f3728037210 legacy=0x7f37280b6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:00.444 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.450+0000 7f3726ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:40:00.444 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.450+0000 7f3726ffd640 1 monclient: found mon.a 2024-03-29T10:40:00.444 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.450+0000 7f3726ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:00.444 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.450+0000 7f3726ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:40:00.444 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.450+0000 7f3726ffd640 1 -- 172.21.15.57:0/3613689929 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37280c6d70 con 0x7f3728033c90 2024-03-29T10:40:00.444 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.450+0000 7f3726ffd640 10 monclient: _finish_auth 0 2024-03-29T10:40:00.444 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.450+0000 7f3726ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:40:00.444 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.450+0000 7f3726ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:00.445 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.450+0000 7f3726ffd640 1 -- 172.21.15.57:0/3613689929 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f373004f730 con 0x7f3728033c90 2024-03-29T10:40:00.445 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.450+0000 7f3726ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:00.445 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.450+0000 7f3734831640 4 set_mon_vals no callback set 2024-03-29T10:40:00.445 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.451+0000 7f3726ffd640 1 -- 172.21.15.57:0/3613689929 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f3730067d90 con 0x7f3728033c90 2024-03-29T10:40:00.445 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.451+0000 7f3726ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:00.445 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.451+0000 7f3726ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:00.445 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.451+0000 7f3726ffd640 10 monclient: dump: 2024-03-29T10:40:00.445 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:00.445 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:00.445 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:00.445 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:00.445 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:00.446 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:00.446 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:00.446 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:00.446 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:00.446 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:00.446 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.451+0000 7f3737036640 5 monclient: authenticate success, global_id 4242 2024-03-29T10:40:00.446 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.451+0000 7f3737036640 10 monclient: _renew_subs 2024-03-29T10:40:00.446 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.451+0000 7f3737036640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:40:00.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.451+0000 7f3737036640 1 -- 172.21.15.57:0/3613689929 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37280cb6e0 con 0x7f3728033c90 2024-03-29T10:40:00.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.451+0000 7f3737036640 10 client.4242.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:00.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.451+0000 7f3737036640 10 monclient: _renew_subs 2024-03-29T10:40:00.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.451+0000 7f3737036640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:40:00.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.451+0000 7f3737036640 1 -- 172.21.15.57:0/3613689929 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37280c8df0 con 0x7f3728033c90 2024-03-29T10:40:00.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.451+0000 7f3737036640 1 librados: init done 2024-03-29T10:40:00.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.453+0000 7f3737036640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:00.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.453+0000 7f3737036640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:00.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.453+0000 7f3737036640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:40:00.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.453+0000 7f3737036640 1 -- 172.21.15.57:0/3613689929 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37280c0050 con 0x7f3728033c90 2024-03-29T10:40:00.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.454+0000 7f3726ffd640 1 -- 172.21.15.57:0/3613689929 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f3730067ef0 con 0x7f3728033c90 2024-03-29T10:40:00.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.455+0000 7f3726ffd640 1 -- 172.21.15.57:0/3613689929 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f37300fdfe0 con 0x7f3728033c90 2024-03-29T10:40:00.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.455+0000 7f3726ffd640 10 client.4242.objecter ms_dispatch 0x7f3730000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:00.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.455+0000 7f3726ffd640 3 client.4242.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:00.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.455+0000 7f3726ffd640 3 client.4242.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:00.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.455+0000 7f3726ffd640 20 client.4242.objecter dump_active .. 0 homeless 2024-03-29T10:40:00.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.455+0000 7f3736034640 10 monclient: get_auth_request con 0x7f371406de50 auth_method 0 2024-03-29T10:40:00.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.456+0000 7f3726ffd640 10 client.4242.objecter ms_handle_connect 0x7f371406de50 2024-03-29T10:40:00.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.463+0000 7f3726ffd640 1 -- 172.21.15.57:0/3613689929 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f37300c98c0 con 0x7f3728033c90 2024-03-29T10:40:00.449 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.463+0000 7f3726ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:00.449 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.463+0000 7f3726ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:00.776 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.792+0000 7f3737036640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:00.776 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.792+0000 7f3737036640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:00.776 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.792+0000 7f3737036640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:40:00.776 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.792+0000 7f3737036640 1 -- 172.21.15.57:0/3613689929 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3728005120 con 0x7f3728033c90 2024-03-29T10:40:00.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.792+0000 7f3726ffd640 1 -- 172.21.15.57:0/3613689929 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f37300cb7f0 con 0x7f3728033c90 2024-03-29T10:40:00.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.792+0000 7f3726ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:00.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.792+0000 7f3726ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:00.778 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:40:00.778 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:00.782 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.796+0000 7f3737036640 10 librados: watch_flush enter 2024-03-29T10:40:00.782 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.796+0000 7f3737036640 10 librados: watch_flush exit 2024-03-29T10:40:00.782 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.796+0000 7f3737036640 20 client.4242.objecter shutdown clearing up homeless session... 2024-03-29T10:40:00.782 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.796+0000 7f3737036640 10 client.4242.objecter successfully canceled tick 2024-03-29T10:40:00.782 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.797+0000 7f3737036640 1 -- 172.21.15.57:0/3613689929 >> v1:172.21.15.179:6816/2698295936 conn(0x7f371406de50 legacy=0x7f3714070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:00.782 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.797+0000 7f3737036640 10 monclient: shutdown 2024-03-29T10:40:00.782 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.797+0000 7f3737036640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:00.782 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.797+0000 7f3737036640 1 -- 172.21.15.57:0/3613689929 >> v1:172.21.15.57:6789/0 conn(0x7f3728033c90 legacy=0x7f37281642f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:00.782 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.797+0000 7f3737036640 1 -- 172.21.15.57:0/3613689929 shutdown_connections 2024-03-29T10:40:00.782 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.797+0000 7f3737036640 1 -- 172.21.15.57:0/3613689929 >> 172.21.15.57:0/3613689929 conn(0x7f372803a550 msgr2=0x7f3728037b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:00.783 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.797+0000 7f3737036640 1 -- 172.21.15.57:0/3613689929 shutdown_connections 2024-03-29T10:40:00.783 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.797+0000 7f3737036640 1 -- 172.21.15.57:0/3613689929 wait complete. 2024-03-29T10:40:00.783 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:00.797+0000 7f3737036640 1 librados: shutdown 2024-03-29T10:40:00.800 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:07.607 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:07.749 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.765+0000 7fb6d09e3640 10 monclient: get_monmap_and_config 2024-03-29T10:40:07.749 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.766+0000 7fb6d09e3640 10 monclient: build_initial_monmap 2024-03-29T10:40:07.749 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.766+0000 7fb6d09e3640 10 monclient: monmap: 2024-03-29T10:40:07.749 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:40:07.750 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:07.750 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:40:07.766742+0000 2024-03-29T10:40:07.750 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:40:07.766742+0000 2024-03-29T10:40:07.750 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:07.750 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:07.750 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:40:07.750 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:40:07.750 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:40:07.750 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:07.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.767+0000 7fb6d09e3640 1 Processor -- start 2024-03-29T10:40:07.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.767+0000 7fb6d09e3640 1 -- start start 2024-03-29T10:40:07.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.767+0000 7fb6d09e3640 10 monclient: init 2024-03-29T10:40:07.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.768+0000 7fb6d09e3640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:07.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.768+0000 7fb6d09e3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:07.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.768+0000 7fb6d09e3640 10 monclient(hunting): picked mon.noname-c con 0x7fb6cc0b2800 addr v1:172.21.15.179:6789/0 2024-03-29T10:40:07.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.768+0000 7fb6d09e3640 10 monclient(hunting): picked mon.noname-b con 0x7fb6cc142b20 addr v1:172.21.15.57:6790/0 2024-03-29T10:40:07.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.768+0000 7fb6d09e3640 10 monclient(hunting): picked mon.noname-a con 0x7fb6cc13c040 addr v1:172.21.15.57:6789/0 2024-03-29T10:40:07.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.768+0000 7fb6d09e3640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6cc0aad10 con 0x7fb6cc13c040 2024-03-29T10:40:07.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.768+0000 7fb6d09e3640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6cc141f30 con 0x7fb6cc0b2800 2024-03-29T10:40:07.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.768+0000 7fb6d09e3640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6cc0aaf60 con 0x7fb6cc142b20 2024-03-29T10:40:07.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.768+0000 7fb6d09e3640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:07.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.768+0000 7fb6d09e3640 10 monclient(hunting): authenticate will time out at 959.944634s 2024-03-29T10:40:07.753 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.769+0000 7fb6cad9d640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fb6cc0b2800 0x7fb6cc142410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:60986/0 (socket says 172.21.15.57:60986) 2024-03-29T10:40:07.753 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.769+0000 7fb6cad9d640 1 -- 172.21.15.57:0/984980672 learned_addr learned my addr 172.21.15.57:0/984980672 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:40:07.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.771+0000 7fb6c9d9b640 1 -- 172.21.15.57:0/984980672 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123164178 0 0) 0x7fb6cc141f30 con 0x7fb6cc0b2800 2024-03-29T10:40:07.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.771+0000 7fb6c9d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:40:07.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.771+0000 7fb6c9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:07.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.771+0000 7fb6c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:07.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.771+0000 7fb6c9d9b640 10 monclient(hunting): my global_id is 4246 2024-03-29T10:40:07.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.771+0000 7fb6c9d9b640 1 -- 172.21.15.57:0/984980672 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6b4003520 con 0x7fb6cc0b2800 2024-03-29T10:40:07.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.771+0000 7fb6c9d9b640 1 -- 172.21.15.57:0/984980672 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 739808107 0 0) 0x7fb6b4003520 con 0x7fb6cc0b2800 2024-03-29T10:40:07.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.771+0000 7fb6c9d9b640 1 -- 172.21.15.57:0/984980672 >> v1:172.21.15.57:6790/0 conn(0x7fb6cc142b20 legacy=0x7fb6cc13b930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:07.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.772+0000 7fb6c9d9b640 1 -- 172.21.15.57:0/984980672 >> v1:172.21.15.57:6789/0 conn(0x7fb6cc13c040 legacy=0x7fb6cc14d7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:07.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.772+0000 7fb6c9d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:40:07.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.772+0000 7fb6c9d9b640 1 monclient: found mon.noname-c 2024-03-29T10:40:07.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.772+0000 7fb6c9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:07.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.772+0000 7fb6c9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:40:07.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.772+0000 7fb6c9d9b640 1 -- 172.21.15.57:0/984980672 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6cc0b3900 con 0x7fb6cc0b2800 2024-03-29T10:40:07.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.772+0000 7fb6c9d9b640 10 monclient: _finish_auth 0 2024-03-29T10:40:07.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.772+0000 7fb6c9d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:40:07.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.772+0000 7fb6c9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:07.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.772+0000 7fb6c9d9b640 1 -- 172.21.15.57:0/984980672 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb6c0001690 con 0x7fb6cc0b2800 2024-03-29T10:40:07.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.772+0000 7fb6c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:07.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.772+0000 7fb6c9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:07.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.772+0000 7fb6c9d9b640 10 monclient: dump: 2024-03-29T10:40:07.757 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:07.757 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:07.757 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:07.757 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:07.757 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:07.757 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:07.757 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:07.758 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:07.758 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:07.758 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:07.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.773+0000 7fb6c9d9b640 1 -- 172.21.15.57:0/984980672 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6c0001860 con 0x7fb6cc0b2800 2024-03-29T10:40:07.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.773+0000 7fb6c9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:07.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.773+0000 7fb6c9d9b640 1 -- 172.21.15.57:0/984980672 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb6c0001cb0 con 0x7fb6cc0b2800 2024-03-29T10:40:07.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.773+0000 7fb6c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:07.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.773+0000 7fb6c9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:07.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.773+0000 7fb6c9d9b640 10 monclient: dump: 2024-03-29T10:40:07.758 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:07.758 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:07.758 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:07.758 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:07.758 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:07.758 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:07.759 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:07.759 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:07.759 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:07.759 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:07.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.773+0000 7fb6d09e3640 5 monclient: authenticate success, global_id 4246 2024-03-29T10:40:07.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.773+0000 7fb6d09e3640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:07.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.773+0000 7fb6d09e3640 4 set_mon_vals no callback set 2024-03-29T10:40:07.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.774+0000 7fb6d09e3640 10 monclient: shutdown 2024-03-29T10:40:07.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.774+0000 7fb6d09e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:07.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.774+0000 7fb6d09e3640 1 -- 172.21.15.57:0/984980672 >> v1:172.21.15.179:6789/0 conn(0x7fb6cc0b2800 legacy=0x7fb6cc142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:07.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.774+0000 7fb6d09e3640 1 -- 172.21.15.57:0/984980672 shutdown_connections 2024-03-29T10:40:07.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.774+0000 7fb6d09e3640 1 -- 172.21.15.57:0/984980672 >> 172.21.15.57:0/984980672 conn(0x7fb6cc06d0e0 msgr2=0x7fb6cc066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:07.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.775+0000 7fb6d09e3640 1 -- 172.21.15.57:0/984980672 shutdown_connections 2024-03-29T10:40:07.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.776+0000 7fb6d09e3640 1 -- 172.21.15.57:0/984980672 wait complete. 2024-03-29T10:40:07.761 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.777+0000 7fb6d09e3640 10 monclient: build_initial_monmap 2024-03-29T10:40:07.761 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.777+0000 7fb6d09e3640 10 monclient: monmap: 2024-03-29T10:40:07.761 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:40:07.761 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:07.761 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:40:07.761 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:40:07.761 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:07.761 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:07.762 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:40:07.762 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:40:07.762 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:40:07.762 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:07.762 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.778+0000 7fb6d09e3640 1 librados: starting msgr at 2024-03-29T10:40:07.762 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.778+0000 7fb6d09e3640 1 librados: starting objecter 2024-03-29T10:40:07.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.778+0000 7fb6d09e3640 1 Processor -- start 2024-03-29T10:40:07.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.778+0000 7fb6d09e3640 1 -- start start 2024-03-29T10:40:07.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.778+0000 7fb6d09e3640 1 librados: setting wanted keys 2024-03-29T10:40:07.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.778+0000 7fb6d09e3640 1 librados: calling monclient init 2024-03-29T10:40:07.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.778+0000 7fb6d09e3640 10 monclient: init 2024-03-29T10:40:07.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.779+0000 7fb6d09e3640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:07.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.779+0000 7fb6d09e3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:07.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.780+0000 7fb6d09e3640 10 monclient(hunting): picked mon.noname-b con 0x7fb6cc0b2800 addr v1:172.21.15.179:6789/0 2024-03-29T10:40:07.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.780+0000 7fb6d09e3640 10 monclient(hunting): picked mon.noname-c con 0x7fb6cc13c040 addr v1:172.21.15.57:6790/0 2024-03-29T10:40:07.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.781+0000 7fb6cad9d640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fb6cc0b2800 0x7fb6cc144750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:33250/0 (socket says 172.21.15.57:33250) 2024-03-29T10:40:07.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.782+0000 7fb6cad9d640 1 -- 172.21.15.57:0/2576059453 learned_addr learned my addr 172.21.15.57:0/2576059453 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:40:07.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.782+0000 7fb6d09e3640 10 monclient(hunting): picked mon.noname-a con 0x7fb6cc142b20 addr v1:172.21.15.57:6789/0 2024-03-29T10:40:07.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.783+0000 7fb6d09e3640 1 -- 172.21.15.57:0/2576059453 --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6cc13d530 con 0x7fb6cc142b20 2024-03-29T10:40:07.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.783+0000 7fb6d09e3640 1 -- 172.21.15.57:0/2576059453 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6cc145d30 con 0x7fb6cc0b2800 2024-03-29T10:40:07.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.783+0000 7fb6d09e3640 1 -- 172.21.15.57:0/2576059453 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6cc144e60 con 0x7fb6cc13c040 2024-03-29T10:40:07.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.783+0000 7fb6d09e3640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:07.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.783+0000 7fb6d09e3640 10 monclient(hunting): authenticate will time out at 959.959310s 2024-03-29T10:40:07.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.783+0000 7fb6b37fe640 10 client.?.objecter ms_handle_connect 0x7fb6cc13c040 2024-03-29T10:40:07.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.783+0000 7fb6b37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:07.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.783+0000 7fb6b37fe640 1 -- 172.21.15.57:0/2576059453 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166748755 0 0) 0x7fb6cc144e60 con 0x7fb6cc13c040 2024-03-29T10:40:07.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.783+0000 7fb6b37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:07.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.783+0000 7fb6b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:07.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.783+0000 7fb6b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:07.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.783+0000 7fb6b37fe640 10 monclient(hunting): my global_id is 4253 2024-03-29T10:40:07.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.783+0000 7fb6b37fe640 1 -- 172.21.15.57:0/2576059453 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6b8004eb0 con 0x7fb6cc13c040 2024-03-29T10:40:07.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.783+0000 7fb6b37fe640 10 client.?.objecter ms_handle_connect 0x7fb6cc0b2800 2024-03-29T10:40:07.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.783+0000 7fb6b37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:07.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.784+0000 7fb6b37fe640 1 -- 172.21.15.57:0/2576059453 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563999470 0 0) 0x7fb6cc145d30 con 0x7fb6cc0b2800 2024-03-29T10:40:07.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.784+0000 7fb6b37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:07.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.784+0000 7fb6b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:07.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.784+0000 7fb6b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:07.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.784+0000 7fb6b37fe640 10 monclient(hunting): my global_id is 4249 2024-03-29T10:40:07.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.784+0000 7fb6b37fe640 1 -- 172.21.15.57:0/2576059453 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6b80051d0 con 0x7fb6cc0b2800 2024-03-29T10:40:07.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.784+0000 7fb6b37fe640 1 -- 172.21.15.57:0/2576059453 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4243486659 0 0) 0x7fb6b80051d0 con 0x7fb6cc0b2800 2024-03-29T10:40:07.775 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.784+0000 7fb6b37fe640 1 -- 172.21.15.57:0/2576059453 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6b8005640 con 0x7fb6cc0b2800 2024-03-29T10:40:07.775 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.784+0000 7fb6b37fe640 1 -- 172.21.15.57:0/2576059453 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb6c0001b70 con 0x7fb6cc0b2800 2024-03-29T10:40:07.775 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.784+0000 7fb6b37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:07.775 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.784+0000 7fb6b37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:07.775 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.784+0000 7fb6b37fe640 10 monclient(hunting): dump: 2024-03-29T10:40:07.775 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:07.775 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:07.775 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:07.775 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:07.775 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:07.775 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:07.775 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:07.775 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:07.775 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:07.775 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:07.776 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.784+0000 7fb6b37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:07.776 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.785+0000 7fb6b37fe640 1 -- 172.21.15.57:0/2576059453 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 878747130 0 0) 0x7fb6b8004eb0 con 0x7fb6cc13c040 2024-03-29T10:40:07.776 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.785+0000 7fb6b37fe640 1 -- 172.21.15.57:0/2576059453 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6b8005cc0 con 0x7fb6cc13c040 2024-03-29T10:40:07.776 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.785+0000 7fb6b37fe640 1 -- 172.21.15.57:0/2576059453 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb6bc001790 con 0x7fb6cc13c040 2024-03-29T10:40:07.776 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.785+0000 7fb6b37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:07.776 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.785+0000 7fb6b37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:07.776 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.785+0000 7fb6b37fe640 10 monclient(hunting): dump: 2024-03-29T10:40:07.776 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:07.776 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:07.776 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:07.776 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:07.776 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:07.776 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:07.776 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:07.776 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:07.777 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:07.777 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:07.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.786+0000 7fb6b37fe640 1 -- 172.21.15.57:0/2576059453 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 431362256 0 0) 0x7fb6b8005640 con 0x7fb6cc0b2800 2024-03-29T10:40:07.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.786+0000 7fb6b37fe640 1 -- 172.21.15.57:0/2576059453 >> v1:172.21.15.57:6790/0 conn(0x7fb6cc13c040 legacy=0x7fb6cc1d6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:07.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.786+0000 7fb6b37fe640 1 -- 172.21.15.57:0/2576059453 >> v1:172.21.15.57:6789/0 conn(0x7fb6cc142b20 legacy=0x7fb6cc1e1720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:40:07.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.786+0000 7fb6b37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:40:07.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.786+0000 7fb6b37fe640 1 monclient: found mon.b 2024-03-29T10:40:07.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.786+0000 7fb6b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:07.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.786+0000 7fb6b37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:40:07.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.786+0000 7fb6b37fe640 1 -- 172.21.15.57:0/2576059453 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6cc144ff0 con 0x7fb6cc0b2800 2024-03-29T10:40:07.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.786+0000 7fb6b37fe640 10 monclient: _finish_auth 0 2024-03-29T10:40:07.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.786+0000 7fb6b37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:40:07.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.786+0000 7fb6b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:07.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.786+0000 7fb6d09e3640 5 monclient: authenticate success, global_id 4249 2024-03-29T10:40:07.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.786+0000 7fb6d09e3640 10 monclient: _renew_subs 2024-03-29T10:40:07.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.786+0000 7fb6d09e3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:40:07.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.786+0000 7fb6d09e3640 1 -- 172.21.15.57:0/2576059453 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6cc1439b0 con 0x7fb6cc0b2800 2024-03-29T10:40:07.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.786+0000 7fb6d09e3640 10 client.4249.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:07.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.786+0000 7fb6d09e3640 10 monclient: _renew_subs 2024-03-29T10:40:07.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.786+0000 7fb6d09e3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:40:07.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.786+0000 7fb6d09e3640 1 -- 172.21.15.57:0/2576059453 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6cc1d78d0 con 0x7fb6cc0b2800 2024-03-29T10:40:07.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.786+0000 7fb6d09e3640 1 librados: init done 2024-03-29T10:40:07.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.787+0000 7fb6b37fe640 1 -- 172.21.15.57:0/2576059453 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6c0004bf0 con 0x7fb6cc0b2800 2024-03-29T10:40:07.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.787+0000 7fb6b37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:07.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.787+0000 7fb6b37fe640 1 -- 172.21.15.57:0/2576059453 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb6c0001b70 con 0x7fb6cc0b2800 2024-03-29T10:40:07.779 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.787+0000 7fb6b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:07.779 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.787+0000 7fb6b37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:07.779 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.787+0000 7fb6b37fe640 10 monclient: dump: 2024-03-29T10:40:07.779 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:07.779 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:07.779 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:07.779 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:07.779 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:07.779 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:07.779 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:07.779 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:07.779 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:07.779 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:07.779 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.788+0000 7fb6d09e3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:07.779 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.788+0000 7fb6d09e3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:07.780 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.788+0000 7fb6d09e3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:40:07.780 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.788+0000 7fb6d09e3640 1 -- 172.21.15.57:0/2576059453 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6cc1d6d00 con 0x7fb6cc0b2800 2024-03-29T10:40:07.780 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.788+0000 7fb6c8d99640 4 set_mon_vals no callback set 2024-03-29T10:40:07.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.790+0000 7fb6b37fe640 1 -- 172.21.15.57:0/2576059453 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fb6c001abd0 con 0x7fb6cc0b2800 2024-03-29T10:40:07.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.791+0000 7fb6b37fe640 1 -- 172.21.15.57:0/2576059453 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7fb6c00848a0 con 0x7fb6cc0b2800 2024-03-29T10:40:07.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.791+0000 7fb6b37fe640 10 client.4249.objecter ms_dispatch 0x7fb6cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:07.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.791+0000 7fb6b37fe640 3 client.4249.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:07.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.791+0000 7fb6b37fe640 3 client.4249.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:07.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.791+0000 7fb6b37fe640 20 client.4249.objecter dump_active .. 0 homeless 2024-03-29T10:40:07.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.792+0000 7fb6ca59c640 10 monclient: get_auth_request con 0x7fb6b806e7c0 auth_method 0 2024-03-29T10:40:07.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.793+0000 7fb6b37fe640 10 client.4249.objecter ms_handle_connect 0x7fb6b806e7c0 2024-03-29T10:40:07.782 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.798+0000 7fb6b37fe640 1 -- 172.21.15.57:0/2576059453 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb6cc1d6d00 con 0x7fb6cc0b2800 2024-03-29T10:40:07.782 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.798+0000 7fb6b37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:07.782 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:07.798+0000 7fb6b37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:08.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:08.120+0000 7fb6d09e3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:08.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:08.120+0000 7fb6d09e3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:08.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:08.120+0000 7fb6d09e3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:40:08.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:08.120+0000 7fb6d09e3640 1 -- 172.21.15.57:0/2576059453 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6cc04f730 con 0x7fb6cc0b2800 2024-03-29T10:40:08.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:08.122+0000 7fb6b37fe640 1 -- 172.21.15.57:0/2576059453 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7fb6c0050180 con 0x7fb6cc0b2800 2024-03-29T10:40:08.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:08.122+0000 7fb6b37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:08.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:08.122+0000 7fb6b37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:08.107 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:40:08.107 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:08.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:08.125+0000 7fb6d09e3640 10 librados: watch_flush enter 2024-03-29T10:40:08.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:08.125+0000 7fb6d09e3640 10 librados: watch_flush exit 2024-03-29T10:40:08.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:08.125+0000 7fb6d09e3640 20 client.4249.objecter shutdown clearing up homeless session... 2024-03-29T10:40:08.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:08.125+0000 7fb6d09e3640 10 client.4249.objecter successfully canceled tick 2024-03-29T10:40:08.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:08.125+0000 7fb6d09e3640 1 -- 172.21.15.57:0/2576059453 >> v1:172.21.15.179:6816/2698295936 conn(0x7fb6b806e7c0 legacy=0x7fb6b8070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:08.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:08.125+0000 7fb6d09e3640 10 monclient: shutdown 2024-03-29T10:40:08.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:08.125+0000 7fb6d09e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:08.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:08.125+0000 7fb6d09e3640 1 -- 172.21.15.57:0/2576059453 >> v1:172.21.15.179:6789/0 conn(0x7fb6cc0b2800 legacy=0x7fb6cc144750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:08.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:08.125+0000 7fb6d09e3640 1 -- 172.21.15.57:0/2576059453 shutdown_connections 2024-03-29T10:40:08.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:08.125+0000 7fb6d09e3640 1 -- 172.21.15.57:0/2576059453 >> 172.21.15.57:0/2576059453 conn(0x7fb6cc06d0e0 msgr2=0x7fb6cc13b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:08.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:08.125+0000 7fb6d09e3640 1 -- 172.21.15.57:0/2576059453 shutdown_connections 2024-03-29T10:40:08.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:08.126+0000 7fb6d09e3640 1 -- 172.21.15.57:0/2576059453 wait complete. 2024-03-29T10:40:08.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:08.126+0000 7fb6d09e3640 1 librados: shutdown 2024-03-29T10:40:08.126 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:14.933 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:15.087 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.104+0000 7f6387713640 10 monclient: get_monmap_and_config 2024-03-29T10:40:15.087 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.104+0000 7f6387713640 10 monclient: build_initial_monmap 2024-03-29T10:40:15.087 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.104+0000 7f6387713640 10 monclient: monmap: 2024-03-29T10:40:15.088 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:40:15.088 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:15.088 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:40:15.104988+0000 2024-03-29T10:40:15.088 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:40:15.104988+0000 2024-03-29T10:40:15.088 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:15.088 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:15.088 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:40:15.088 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:40:15.088 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:40:15.088 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:15.089 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.105+0000 7f6387713640 1 Processor -- start 2024-03-29T10:40:15.089 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.105+0000 7f6387713640 1 -- start start 2024-03-29T10:40:15.089 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.105+0000 7f6387713640 10 monclient: init 2024-03-29T10:40:15.091 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.106+0000 7f6387713640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:15.091 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.106+0000 7f6387713640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:40:15.091 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.108+0000 7f6387713640 10 monclient(hunting): picked mon.noname-b con 0x7f63801457f0 addr v1:172.21.15.57:6790/0 2024-03-29T10:40:15.092 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.108+0000 7f6387713640 10 monclient(hunting): picked mon.noname-a con 0x7f638013f650 addr v1:172.21.15.57:6789/0 2024-03-29T10:40:15.092 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.108+0000 7f6387713640 10 monclient(hunting): picked mon.noname-c con 0x7f63800a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T10:40:15.092 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.108+0000 7f6387713640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63800aaef0 con 0x7f638013f650 2024-03-29T10:40:15.092 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.108+0000 7f6387713640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6380145bd0 con 0x7f63800a9d70 2024-03-29T10:40:15.092 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.108+0000 7f6387713640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6380145eb0 con 0x7f63801457f0 2024-03-29T10:40:15.092 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.108+0000 7f6387713640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:15.092 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.108+0000 7f6387713640 10 monclient(hunting): authenticate will time out at 967.284409s 2024-03-29T10:40:15.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.110+0000 7f63854af640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f63801457f0 0x7f638013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:51353/0 (socket says 172.21.15.57:51353) 2024-03-29T10:40:15.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.110+0000 7f63854af640 1 -- 172.21.15.57:0/3384557376 learned_addr learned my addr 172.21.15.57:0/3384557376 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:40:15.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.111+0000 7f6377fff640 1 -- 172.21.15.57:0/3384557376 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656491410 0 0) 0x7f6380145eb0 con 0x7f63801457f0 2024-03-29T10:40:15.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.111+0000 7f6377fff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:15.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.111+0000 7f6377fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:15.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.111+0000 7f6377fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:15.095 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.111+0000 7f6377fff640 10 monclient(hunting): my global_id is 4256 2024-03-29T10:40:15.095 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.111+0000 7f6377fff640 1 -- 172.21.15.57:0/3384557376 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f636c003520 con 0x7f63801457f0 2024-03-29T10:40:15.095 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.111+0000 7f6377fff640 1 -- 172.21.15.57:0/3384557376 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 214060208 0 0) 0x7f636c003520 con 0x7f63801457f0 2024-03-29T10:40:15.095 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.111+0000 7f6377fff640 1 -- 172.21.15.57:0/3384557376 >> v1:172.21.15.179:6789/0 conn(0x7f63800a9d70 legacy=0x7f638014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:15.095 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.112+0000 7f6377fff640 1 -- 172.21.15.57:0/3384557376 >> v1:172.21.15.57:6789/0 conn(0x7f638013f650 legacy=0x7f63800b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:15.096 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.112+0000 7f6377fff640 10 monclient: _finish_hunting 0 2024-03-29T10:40:15.096 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.112+0000 7f6377fff640 1 monclient: found mon.noname-b 2024-03-29T10:40:15.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.112+0000 7f6377fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:15.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.112+0000 7f6377fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:40:15.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.112+0000 7f6377fff640 1 -- 172.21.15.57:0/3384557376 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6380146340 con 0x7f63801457f0 2024-03-29T10:40:15.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.112+0000 7f6377fff640 10 monclient: _finish_auth 0 2024-03-29T10:40:15.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.112+0000 7f6377fff640 10 monclient: _check_auth_tickets 2024-03-29T10:40:15.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.112+0000 7f6377fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:15.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.112+0000 7f6377fff640 1 -- 172.21.15.57:0/3384557376 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f637c001690 con 0x7f63801457f0 2024-03-29T10:40:15.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.112+0000 7f6377fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:15.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.113+0000 7f6377fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:15.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.113+0000 7f6377fff640 10 monclient: dump: 2024-03-29T10:40:15.097 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:15.097 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:15.098 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:15.098 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:15.098 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:15.098 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:15.098 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:15.098 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:15.098 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:15.098 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:15.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.113+0000 7f6387713640 5 monclient: authenticate success, global_id 4256 2024-03-29T10:40:15.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.113+0000 7f6387713640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:15.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.113+0000 7f6377fff640 1 -- 172.21.15.57:0/3384557376 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f637c001920 con 0x7f63801457f0 2024-03-29T10:40:15.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.113+0000 7f6377fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:15.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.113+0000 7f6377fff640 1 -- 172.21.15.57:0/3384557376 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f637c001cb0 con 0x7f63801457f0 2024-03-29T10:40:15.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.113+0000 7f6387713640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:15.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.113+0000 7f6387713640 4 set_mon_vals no callback set 2024-03-29T10:40:15.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.113+0000 7f6377fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:15.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.113+0000 7f6377fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:15.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.113+0000 7f6377fff640 10 monclient: dump: 2024-03-29T10:40:15.099 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:15.099 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:15.099 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:15.099 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:15.099 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:15.099 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:15.099 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:15.099 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:15.099 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:15.100 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:15.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.114+0000 7f6387713640 10 monclient: shutdown 2024-03-29T10:40:15.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.114+0000 7f6387713640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:15.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.114+0000 7f6387713640 1 -- 172.21.15.57:0/3384557376 >> v1:172.21.15.57:6790/0 conn(0x7f63801457f0 legacy=0x7f638013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:15.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.117+0000 7f6387713640 1 -- 172.21.15.57:0/3384557376 shutdown_connections 2024-03-29T10:40:15.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.117+0000 7f6387713640 1 -- 172.21.15.57:0/3384557376 >> 172.21.15.57:0/3384557376 conn(0x7f6380066a00 msgr2=0x7f6380066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:15.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.117+0000 7f6387713640 1 -- 172.21.15.57:0/3384557376 shutdown_connections 2024-03-29T10:40:15.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.117+0000 7f6387713640 1 -- 172.21.15.57:0/3384557376 wait complete. 2024-03-29T10:40:15.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.118+0000 7f6387713640 10 monclient: build_initial_monmap 2024-03-29T10:40:15.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.118+0000 7f6387713640 10 monclient: monmap: 2024-03-29T10:40:15.102 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:40:15.102 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:15.102 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:40:15.102 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:40:15.102 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:15.103 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:15.103 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:40:15.103 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:40:15.103 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:40:15.103 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:15.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.119+0000 7f6387713640 1 librados: starting msgr at 2024-03-29T10:40:15.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.119+0000 7f6387713640 1 librados: starting objecter 2024-03-29T10:40:15.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.119+0000 7f6387713640 1 Processor -- start 2024-03-29T10:40:15.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.119+0000 7f6387713640 1 -- start start 2024-03-29T10:40:15.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.119+0000 7f6387713640 1 librados: setting wanted keys 2024-03-29T10:40:15.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.119+0000 7f6387713640 1 librados: calling monclient init 2024-03-29T10:40:15.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.119+0000 7f6387713640 10 monclient: init 2024-03-29T10:40:15.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.121+0000 7f6387713640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:15.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.121+0000 7f6387713640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:40:15.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.121+0000 7f6387713640 10 monclient(hunting): picked mon.noname-c con 0x7f63800a9d70 addr v1:172.21.15.57:6790/0 2024-03-29T10:40:15.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.121+0000 7f6387713640 10 monclient(hunting): picked mon.noname-b con 0x7f638013f650 addr v1:172.21.15.179:6789/0 2024-03-29T10:40:15.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.121+0000 7f6387713640 10 monclient(hunting): picked mon.noname-a con 0x7f63801457f0 addr v1:172.21.15.57:6789/0 2024-03-29T10:40:15.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.121+0000 7f6387713640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6380135750 con 0x7f63801457f0 2024-03-29T10:40:15.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.121+0000 7f6387713640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63801359a0 con 0x7f638013f650 2024-03-29T10:40:15.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.121+0000 7f6387713640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63801417e0 con 0x7f63800a9d70 2024-03-29T10:40:15.106 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.121+0000 7f6387713640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:15.106 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.121+0000 7f6387713640 10 monclient(hunting): authenticate will time out at 967.297924s 2024-03-29T10:40:15.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.127+0000 7f6384cae640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f638013f650 0x7f63801db720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:43989/0 (socket says 172.21.15.57:43989) 2024-03-29T10:40:15.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.127+0000 7f6384cae640 1 -- 172.21.15.57:0/2675713930 learned_addr learned my addr 172.21.15.57:0/2675713930 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:40:15.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.128+0000 7f6375ffb640 10 client.?.objecter ms_handle_connect 0x7f638013f650 2024-03-29T10:40:15.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.128+0000 7f6375ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:15.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.128+0000 7f6375ffb640 1 -- 172.21.15.57:0/2675713930 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607112717 0 0) 0x7f63801359a0 con 0x7f638013f650 2024-03-29T10:40:15.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.128+0000 7f6375ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:15.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.128+0000 7f6375ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:15.113 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.128+0000 7f6375ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:15.113 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.128+0000 7f6375ffb640 10 monclient(hunting): my global_id is 4252 2024-03-29T10:40:15.113 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.128+0000 7f6375ffb640 1 -- 172.21.15.57:0/2675713930 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6364004eb0 con 0x7f638013f650 2024-03-29T10:40:15.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.130+0000 7f6375ffb640 10 client.?.objecter ms_handle_connect 0x7f63800a9d70 2024-03-29T10:40:15.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.130+0000 7f6375ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:15.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.130+0000 7f6375ffb640 10 client.?.objecter ms_handle_connect 0x7f63801457f0 2024-03-29T10:40:15.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.130+0000 7f6375ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:15.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.130+0000 7f6375ffb640 1 -- 172.21.15.57:0/2675713930 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 107019463 0 0) 0x7f6364004eb0 con 0x7f638013f650 2024-03-29T10:40:15.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.130+0000 7f6375ffb640 1 -- 172.21.15.57:0/2675713930 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63801359a0 con 0x7f638013f650 2024-03-29T10:40:15.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.130+0000 7f6375ffb640 1 -- 172.21.15.57:0/2675713930 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657344845 0 0) 0x7f6380135750 con 0x7f63801457f0 2024-03-29T10:40:15.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.131+0000 7f6375ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:15.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.131+0000 7f6375ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:15.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.131+0000 7f6375ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:15.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.131+0000 7f6375ffb640 10 monclient(hunting): my global_id is 4245 2024-03-29T10:40:15.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.131+0000 7f6375ffb640 1 -- 172.21.15.57:0/2675713930 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6364004eb0 con 0x7f63801457f0 2024-03-29T10:40:15.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.131+0000 7f6375ffb640 1 -- 172.21.15.57:0/2675713930 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3437273230 0 0) 0x7f63801417e0 con 0x7f63800a9d70 2024-03-29T10:40:15.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.131+0000 7f6375ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:15.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.131+0000 7f6375ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:15.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.131+0000 7f6375ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:15.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.131+0000 7f6375ffb640 10 monclient(hunting): my global_id is 4259 2024-03-29T10:40:15.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.131+0000 7f6375ffb640 1 -- 172.21.15.57:0/2675713930 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6380135750 con 0x7f63800a9d70 2024-03-29T10:40:15.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.131+0000 7f6375ffb640 1 -- 172.21.15.57:0/2675713930 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f6370001960 con 0x7f638013f650 2024-03-29T10:40:15.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.131+0000 7f6375ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:15.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.132+0000 7f6375ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:15.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.132+0000 7f6375ffb640 10 monclient(hunting): dump: 2024-03-29T10:40:15.120 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:15.120 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:15.120 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:15.120 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:15.120 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:15.120 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:15.120 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:15.120 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:15.120 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:15.120 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:15.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.132+0000 7f6375ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:15.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.132+0000 7f6375ffb640 1 -- 172.21.15.57:0/2675713930 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1728179758 0 0) 0x7f63801359a0 con 0x7f638013f650 2024-03-29T10:40:15.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.132+0000 7f6375ffb640 1 -- 172.21.15.57:0/2675713930 >> v1:172.21.15.57:6790/0 conn(0x7f63800a9d70 legacy=0x7f638013b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:15.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.132+0000 7f6375ffb640 1 -- 172.21.15.57:0/2675713930 >> v1:172.21.15.57:6789/0 conn(0x7f63801457f0 legacy=0x7f63801deb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:15.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.132+0000 7f6375ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:40:15.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.132+0000 7f6375ffb640 1 monclient: found mon.b 2024-03-29T10:40:15.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.132+0000 7f6375ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:15.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.132+0000 7f6375ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:40:15.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.132+0000 7f6375ffb640 1 -- 172.21.15.57:0/2675713930 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638013fda0 con 0x7f638013f650 2024-03-29T10:40:15.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.132+0000 7f6375ffb640 10 monclient: _finish_auth 0 2024-03-29T10:40:15.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.132+0000 7f6375ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:40:15.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.132+0000 7f6375ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:15.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.132+0000 7f6387713640 5 monclient: authenticate success, global_id 4252 2024-03-29T10:40:15.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.132+0000 7f6387713640 10 monclient: _renew_subs 2024-03-29T10:40:15.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.132+0000 7f6387713640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:40:15.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.132+0000 7f6387713640 1 -- 172.21.15.57:0/2675713930 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63801dbe30 con 0x7f638013f650 2024-03-29T10:40:15.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.133+0000 7f6387713640 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:15.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.133+0000 7f6387713640 10 monclient: _renew_subs 2024-03-29T10:40:15.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.133+0000 7f6387713640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:40:15.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.133+0000 7f6387713640 1 -- 172.21.15.57:0/2675713930 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63801df280 con 0x7f638013f650 2024-03-29T10:40:15.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.133+0000 7f6387713640 1 librados: init done 2024-03-29T10:40:15.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.133+0000 7f6375ffb640 1 -- 172.21.15.57:0/2675713930 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6370001ac0 con 0x7f638013f650 2024-03-29T10:40:15.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.133+0000 7f6375ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:15.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.133+0000 7f6375ffb640 1 -- 172.21.15.57:0/2675713930 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f6370001d80 con 0x7f638013f650 2024-03-29T10:40:15.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.133+0000 7f6375ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:15.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.133+0000 7f6375ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:15.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.133+0000 7f6375ffb640 10 monclient: dump: 2024-03-29T10:40:15.122 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:15.123 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:15.123 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:15.123 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:15.123 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:15.123 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:15.123 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:15.123 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:15.123 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:15.123 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:15.123 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.133+0000 7f6376ffd640 4 set_mon_vals no callback set 2024-03-29T10:40:15.124 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.135+0000 7f6387713640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:15.124 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.135+0000 7f6387713640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:15.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.136+0000 7f6387713640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:40:15.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.136+0000 7f6387713640 1 -- 172.21.15.57:0/2675713930 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6360005080 con 0x7f638013f650 2024-03-29T10:40:15.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.137+0000 7f6375ffb640 1 -- 172.21.15.57:0/2675713930 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f6370003b40 con 0x7f638013f650 2024-03-29T10:40:15.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.138+0000 7f6375ffb640 1 -- 172.21.15.57:0/2675713930 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f6370084a50 con 0x7f638013f650 2024-03-29T10:40:15.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.139+0000 7f6375ffb640 10 client.4252.objecter ms_dispatch 0x7f6380000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:15.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.139+0000 7f63854af640 10 monclient: get_auth_request con 0x7f636406dd30 auth_method 0 2024-03-29T10:40:15.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.139+0000 7f6375ffb640 3 client.4252.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:15.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.139+0000 7f6375ffb640 3 client.4252.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:15.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.139+0000 7f6375ffb640 20 client.4252.objecter dump_active .. 0 homeless 2024-03-29T10:40:15.126 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.143+0000 7f6375ffb640 10 client.4252.objecter ms_handle_connect 0x7f636406dd30 2024-03-29T10:40:15.130 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.146+0000 7f6375ffb640 1 -- 172.21.15.57:0/2675713930 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6370050330 con 0x7f638013f650 2024-03-29T10:40:15.130 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.146+0000 7f6375ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:15.131 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.147+0000 7f6375ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:15.462 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.478+0000 7f6387713640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:15.462 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.478+0000 7f6387713640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:15.462 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.478+0000 7f6387713640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:40:15.462 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.478+0000 7f6387713640 1 -- 172.21.15.57:0/2675713930 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6360005330 con 0x7f638013f650 2024-03-29T10:40:15.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.479+0000 7f6375ffb640 1 -- 172.21.15.57:0/2675713930 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f6370052260 con 0x7f638013f650 2024-03-29T10:40:15.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.479+0000 7f6375ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:15.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.479+0000 7f6375ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:15.463 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:40:15.463 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:15.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.483+0000 7f6387713640 10 librados: watch_flush enter 2024-03-29T10:40:15.468 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.485+0000 7f6387713640 10 librados: watch_flush exit 2024-03-29T10:40:15.468 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.485+0000 7f6387713640 20 client.4252.objecter shutdown clearing up homeless session... 2024-03-29T10:40:15.468 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.485+0000 7f6387713640 10 client.4252.objecter successfully canceled tick 2024-03-29T10:40:15.469 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.485+0000 7f6387713640 1 -- 172.21.15.57:0/2675713930 >> v1:172.21.15.179:6816/2698295936 conn(0x7f636406dd30 legacy=0x7f63640701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:15.469 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.485+0000 7f6387713640 10 monclient: shutdown 2024-03-29T10:40:15.469 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.485+0000 7f6387713640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:15.469 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.485+0000 7f6387713640 1 -- 172.21.15.57:0/2675713930 >> v1:172.21.15.179:6789/0 conn(0x7f638013f650 legacy=0x7f63801db720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:15.469 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.485+0000 7f6387713640 1 -- 172.21.15.57:0/2675713930 shutdown_connections 2024-03-29T10:40:15.469 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.485+0000 7f6387713640 1 -- 172.21.15.57:0/2675713930 >> 172.21.15.57:0/2675713930 conn(0x7f6380066a00 msgr2=0x7f63800b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:15.469 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.485+0000 7f6387713640 1 -- 172.21.15.57:0/2675713930 shutdown_connections 2024-03-29T10:40:15.469 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.485+0000 7f6387713640 1 -- 172.21.15.57:0/2675713930 wait complete. 2024-03-29T10:40:15.470 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:15.486+0000 7f6387713640 1 librados: shutdown 2024-03-29T10:40:15.485 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:22.293 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:22.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.464+0000 7f3908f10640 10 monclient: get_monmap_and_config 2024-03-29T10:40:22.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.464+0000 7f3908f10640 10 monclient: build_initial_monmap 2024-03-29T10:40:22.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.464+0000 7f3908f10640 10 monclient: monmap: 2024-03-29T10:40:22.448 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:40:22.448 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:22.448 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:40:22.465003+0000 2024-03-29T10:40:22.448 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:40:22.465003+0000 2024-03-29T10:40:22.448 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:22.448 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:22.448 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:40:22.448 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:40:22.448 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:40:22.448 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:22.450 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.467+0000 7f3908f10640 1 Processor -- start 2024-03-29T10:40:22.451 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.467+0000 7f3908f10640 1 -- start start 2024-03-29T10:40:22.451 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.467+0000 7f3908f10640 10 monclient: init 2024-03-29T10:40:22.451 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.468+0000 7f3908f10640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:22.451 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.468+0000 7f3908f10640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:22.451 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.468+0000 7f3908f10640 10 monclient(hunting): picked mon.noname-c con 0x7f38fc038c30 addr v1:172.21.15.179:6789/0 2024-03-29T10:40:22.451 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.468+0000 7f3908f10640 10 monclient(hunting): picked mon.noname-a con 0x7f38fc032480 addr v1:172.21.15.57:6789/0 2024-03-29T10:40:22.451 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.468+0000 7f3908f10640 10 monclient(hunting): picked mon.noname-b con 0x7f38fc0d11b0 addr v1:172.21.15.57:6790/0 2024-03-29T10:40:22.451 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.468+0000 7f3908f10640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38fc0caf50 con 0x7f38fc032480 2024-03-29T10:40:22.451 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.468+0000 7f3908f10640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38fc039820 con 0x7f38fc038c30 2024-03-29T10:40:22.451 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.468+0000 7f3908f10640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38fc0c78a0 con 0x7f38fc0d11b0 2024-03-29T10:40:22.452 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.468+0000 7f3908f10640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:22.452 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.468+0000 7f3908f10640 10 monclient(hunting): authenticate will time out at 974.644184s 2024-03-29T10:40:22.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.471+0000 7f3903fff640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f38fc0d11b0 0x7f38fc0d35a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:50051/0 (socket says 172.21.15.57:50051) 2024-03-29T10:40:22.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.471+0000 7f3903fff640 1 -- 172.21.15.57:0/946548630 learned_addr learned my addr 172.21.15.57:0/946548630 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:40:22.455 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.471+0000 7f39027fc640 1 -- 172.21.15.57:0/946548630 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263204876 0 0) 0x7f38fc0c78a0 con 0x7f38fc0d11b0 2024-03-29T10:40:22.455 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.471+0000 7f39027fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:22.455 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.471+0000 7f39027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:22.455 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.471+0000 7f39027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:22.455 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.471+0000 7f39027fc640 10 monclient(hunting): my global_id is 4262 2024-03-29T10:40:22.455 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.472+0000 7f39027fc640 1 -- 172.21.15.57:0/946548630 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38f4003520 con 0x7f38fc0d11b0 2024-03-29T10:40:22.460 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.475+0000 7f39027fc640 1 -- 172.21.15.57:0/946548630 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2993096874 0 0) 0x7f38f4003520 con 0x7f38fc0d11b0 2024-03-29T10:40:22.460 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.475+0000 7f39027fc640 1 -- 172.21.15.57:0/946548630 >> v1:172.21.15.179:6789/0 conn(0x7f38fc038c30 legacy=0x7f38fc0c4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:40:22.460 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.475+0000 7f39027fc640 1 -- 172.21.15.57:0/946548630 >> v1:172.21.15.57:6789/0 conn(0x7f38fc032480 legacy=0x7f38fc0cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:22.460 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.475+0000 7f39027fc640 10 monclient: _finish_hunting 0 2024-03-29T10:40:22.460 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.475+0000 7f39027fc640 1 monclient: found mon.noname-b 2024-03-29T10:40:22.460 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.475+0000 7f39027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:22.460 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.475+0000 7f39027fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:40:22.461 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.475+0000 7f39027fc640 1 -- 172.21.15.57:0/946548630 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38fc0c7a30 con 0x7f38fc0d11b0 2024-03-29T10:40:22.461 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.475+0000 7f39027fc640 10 monclient: _finish_auth 0 2024-03-29T10:40:22.461 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.475+0000 7f39027fc640 10 monclient: _check_auth_tickets 2024-03-29T10:40:22.461 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.475+0000 7f39027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:22.461 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.475+0000 7f3908f10640 5 monclient: authenticate success, global_id 4262 2024-03-29T10:40:22.461 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.475+0000 7f3908f10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:22.461 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.475+0000 7f39027fc640 1 -- 172.21.15.57:0/946548630 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f38f8001690 con 0x7f38fc0d11b0 2024-03-29T10:40:22.461 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.475+0000 7f39027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:22.461 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.475+0000 7f39027fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:22.461 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.475+0000 7f39027fc640 10 monclient: dump: 2024-03-29T10:40:22.461 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:22.461 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:22.461 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:22.461 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:22.461 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:22.462 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:22.462 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:22.462 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:22.462 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:22.462 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:22.462 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.475+0000 7f3908f10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:22.462 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.476+0000 7f39027fc640 1 -- 172.21.15.57:0/946548630 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38f8001860 con 0x7f38fc0d11b0 2024-03-29T10:40:22.462 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.476+0000 7f39027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:22.462 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.476+0000 7f39027fc640 1 -- 172.21.15.57:0/946548630 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f38f8001cb0 con 0x7f38fc0d11b0 2024-03-29T10:40:22.462 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.476+0000 7f3908f10640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:22.462 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.476+0000 7f3908f10640 4 set_mon_vals no callback set 2024-03-29T10:40:22.462 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.476+0000 7f39027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:22.462 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.476+0000 7f39027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:22.462 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.476+0000 7f39027fc640 10 monclient: dump: 2024-03-29T10:40:22.463 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:22.463 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:22.463 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:22.463 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:22.463 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:22.463 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:22.463 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:22.463 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:22.463 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:22.463 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:22.464 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.477+0000 7f3908f10640 10 monclient: shutdown 2024-03-29T10:40:22.464 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.477+0000 7f3908f10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:22.464 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.477+0000 7f3908f10640 1 -- 172.21.15.57:0/946548630 >> v1:172.21.15.57:6790/0 conn(0x7f38fc0d11b0 legacy=0x7f38fc0d35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:22.464 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.477+0000 7f3908f10640 1 -- 172.21.15.57:0/946548630 shutdown_connections 2024-03-29T10:40:22.464 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.477+0000 7f3908f10640 1 -- 172.21.15.57:0/946548630 >> 172.21.15.57:0/946548630 conn(0x7f38fc005bf0 msgr2=0x7f38fc006000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:22.464 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.480+0000 7f3908f10640 1 -- 172.21.15.57:0/946548630 shutdown_connections 2024-03-29T10:40:22.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.481+0000 7f3908f10640 1 -- 172.21.15.57:0/946548630 wait complete. 2024-03-29T10:40:22.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.482+0000 7f3908f10640 10 monclient: build_initial_monmap 2024-03-29T10:40:22.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.482+0000 7f3908f10640 10 monclient: monmap: 2024-03-29T10:40:22.466 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:40:22.466 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:22.466 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:40:22.466 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:40:22.466 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:22.467 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:22.467 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:40:22.467 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:40:22.467 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:40:22.467 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:22.469 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.483+0000 7f3908f10640 1 librados: starting msgr at 2024-03-29T10:40:22.469 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.483+0000 7f3908f10640 1 librados: starting objecter 2024-03-29T10:40:22.469 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.483+0000 7f3908f10640 1 Processor -- start 2024-03-29T10:40:22.469 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.483+0000 7f3908f10640 1 -- start start 2024-03-29T10:40:22.469 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.483+0000 7f3908f10640 1 librados: setting wanted keys 2024-03-29T10:40:22.469 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.483+0000 7f3908f10640 1 librados: calling monclient init 2024-03-29T10:40:22.469 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.483+0000 7f3908f10640 10 monclient: init 2024-03-29T10:40:22.469 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.484+0000 7f3908f10640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:22.469 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.484+0000 7f3908f10640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:22.470 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.484+0000 7f3908f10640 10 monclient(hunting): picked mon.noname-b con 0x7f38fc032480 addr v1:172.21.15.179:6789/0 2024-03-29T10:40:22.470 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.484+0000 7f3908f10640 10 monclient(hunting): picked mon.noname-c con 0x7f38fc038c30 addr v1:172.21.15.57:6790/0 2024-03-29T10:40:22.470 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.484+0000 7f3908f10640 10 monclient(hunting): picked mon.noname-a con 0x7f38fc0d11b0 addr v1:172.21.15.57:6789/0 2024-03-29T10:40:22.470 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.484+0000 7f3908f10640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38fc0365f0 con 0x7f38fc0d11b0 2024-03-29T10:40:22.470 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.484+0000 7f3908f10640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38fc036860 con 0x7f38fc032480 2024-03-29T10:40:22.470 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.484+0000 7f3908f10640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38fc036b40 con 0x7f38fc038c30 2024-03-29T10:40:22.470 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.484+0000 7f3908f10640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:22.470 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.484+0000 7f3908f10640 10 monclient(hunting): authenticate will time out at 974.660762s 2024-03-29T10:40:22.471 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.487+0000 7f3902ffd640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f38fc038c30 0x7f38fc0bc8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:50161/0 (socket says 172.21.15.57:50161) 2024-03-29T10:40:22.471 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.487+0000 7f3902ffd640 1 -- 172.21.15.57:0/4221829994 learned_addr learned my addr 172.21.15.57:0/4221829994 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:40:22.471 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.487+0000 7f38effff640 10 client.?.objecter ms_handle_connect 0x7f38fc038c30 2024-03-29T10:40:22.471 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.487+0000 7f38effff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:22.472 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.488+0000 7f38effff640 1 -- 172.21.15.57:0/4221829994 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897772851 0 0) 0x7f38fc036b40 con 0x7f38fc038c30 2024-03-29T10:40:22.472 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.488+0000 7f38effff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:22.472 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.488+0000 7f38effff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:22.472 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.488+0000 7f38effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:22.472 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.488+0000 7f38effff640 10 monclient(hunting): my global_id is 4265 2024-03-29T10:40:22.472 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.488+0000 7f38effff640 1 -- 172.21.15.57:0/4221829994 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38e8004eb0 con 0x7f38fc038c30 2024-03-29T10:40:22.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.489+0000 7f38effff640 1 -- 172.21.15.57:0/4221829994 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 550536015 0 0) 0x7f38e8004eb0 con 0x7f38fc038c30 2024-03-29T10:40:22.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.489+0000 7f38effff640 1 -- 172.21.15.57:0/4221829994 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38fc036b40 con 0x7f38fc038c30 2024-03-29T10:40:22.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.489+0000 7f38effff640 1 -- 172.21.15.57:0/4221829994 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f38f0001a30 con 0x7f38fc038c30 2024-03-29T10:40:22.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.489+0000 7f38effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:22.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.489+0000 7f38effff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:22.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.489+0000 7f38effff640 10 monclient(hunting): dump: 2024-03-29T10:40:22.473 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:22.473 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:22.473 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:22.473 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:22.473 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:22.473 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:22.473 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:22.474 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:22.474 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:22.474 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:22.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.489+0000 7f38effff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:22.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.490+0000 7f38effff640 1 -- 172.21.15.57:0/4221829994 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4064710327 0 0) 0x7f38fc036b40 con 0x7f38fc038c30 2024-03-29T10:40:22.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.490+0000 7f38effff640 1 -- 172.21.15.57:0/4221829994 >> v1:172.21.15.179:6789/0 conn(0x7f38fc032480 legacy=0x7f38fc0354d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:22.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.490+0000 7f38effff640 1 -- 172.21.15.57:0/4221829994 >> v1:172.21.15.57:6789/0 conn(0x7f38fc0d11b0 legacy=0x7f38fc0bcff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:40:22.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.491+0000 7f38effff640 10 monclient: _finish_hunting 0 2024-03-29T10:40:22.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.491+0000 7f38effff640 1 monclient: found mon.c 2024-03-29T10:40:22.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.491+0000 7f38effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:22.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.491+0000 7f38effff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:40:22.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.491+0000 7f38effff640 1 -- 172.21.15.57:0/4221829994 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38fc0343c0 con 0x7f38fc038c30 2024-03-29T10:40:22.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.491+0000 7f38effff640 10 monclient: _finish_auth 0 2024-03-29T10:40:22.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.491+0000 7f38effff640 10 monclient: _check_auth_tickets 2024-03-29T10:40:22.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.491+0000 7f38effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:22.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.491+0000 7f3908f10640 5 monclient: authenticate success, global_id 4265 2024-03-29T10:40:22.477 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.491+0000 7f3908f10640 10 monclient: _renew_subs 2024-03-29T10:40:22.477 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.491+0000 7f3908f10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:40:22.477 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.491+0000 7f3908f10640 1 -- 172.21.15.57:0/4221829994 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38fc0bae60 con 0x7f38fc038c30 2024-03-29T10:40:22.477 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.491+0000 7f3908f10640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:22.477 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.491+0000 7f3908f10640 10 monclient: _renew_subs 2024-03-29T10:40:22.477 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.491+0000 7f3908f10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:40:22.477 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.491+0000 7f3908f10640 1 -- 172.21.15.57:0/4221829994 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38fc0bb2f0 con 0x7f38fc038c30 2024-03-29T10:40:22.477 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.491+0000 7f3908f10640 1 librados: init done 2024-03-29T10:40:22.477 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.492+0000 7f3908f10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:22.477 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.493+0000 7f3908f10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:22.478 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.493+0000 7f3908f10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:40:22.478 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.493+0000 7f3908f10640 1 -- 172.21.15.57:0/4221829994 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38fc007b60 con 0x7f38fc038c30 2024-03-29T10:40:22.478 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.495+0000 7f38effff640 1 -- 172.21.15.57:0/4221829994 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38f0001b90 con 0x7f38fc038c30 2024-03-29T10:40:22.479 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.495+0000 7f38effff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:22.479 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.495+0000 7f38effff640 1 -- 172.21.15.57:0/4221829994 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f38f0003a60 con 0x7f38fc038c30 2024-03-29T10:40:22.479 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.495+0000 7f38effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:22.479 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.495+0000 7f38effff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:22.479 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.495+0000 7f38effff640 10 monclient: dump: 2024-03-29T10:40:22.479 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:22.479 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:22.479 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:22.479 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:22.479 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:22.479 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:22.479 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:22.479 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:22.480 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:22.480 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:22.480 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.495+0000 7f3901ffb640 4 set_mon_vals no callback set 2024-03-29T10:40:22.480 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.496+0000 7f38effff640 1 -- 172.21.15.57:0/4221829994 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f38f0003c80 con 0x7f38fc038c30 2024-03-29T10:40:22.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.497+0000 7f38effff640 1 -- 172.21.15.57:0/4221829994 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f38f0084bb0 con 0x7f38fc038c30 2024-03-29T10:40:22.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.497+0000 7f38effff640 10 client.4265.objecter ms_dispatch 0x7f3904000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:22.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.497+0000 7f39037fe640 10 monclient: get_auth_request con 0x7f38e806dcc0 auth_method 0 2024-03-29T10:40:22.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.497+0000 7f38effff640 3 client.4265.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:22.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.498+0000 7f38effff640 3 client.4265.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:22.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.498+0000 7f38effff640 20 client.4265.objecter dump_active .. 0 homeless 2024-03-29T10:40:22.482 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.498+0000 7f38effff640 10 client.4265.objecter ms_handle_connect 0x7f38e806dcc0 2024-03-29T10:40:22.488 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.504+0000 7f38effff640 1 -- 172.21.15.57:0/4221829994 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f38f00505f0 con 0x7f38fc038c30 2024-03-29T10:40:22.488 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.504+0000 7f38effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:22.488 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.504+0000 7f38effff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:22.819 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.835+0000 7f3908f10640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:22.819 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.836+0000 7f3908f10640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:22.820 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.836+0000 7f3908f10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:40:22.820 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.836+0000 7f3908f10640 1 -- 172.21.15.57:0/4221829994 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38fc037060 con 0x7f38fc038c30 2024-03-29T10:40:22.821 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.837+0000 7f38effff640 1 -- 172.21.15.57:0/4221829994 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f38fc037060 con 0x7f38fc038c30 2024-03-29T10:40:22.821 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.837+0000 7f38effff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:22.821 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.837+0000 7f38effff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:22.821 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:40:22.821 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:22.825 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.841+0000 7f3908f10640 10 librados: watch_flush enter 2024-03-29T10:40:22.825 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.842+0000 7f3908f10640 10 librados: watch_flush exit 2024-03-29T10:40:22.825 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.842+0000 7f3908f10640 20 client.4265.objecter shutdown clearing up homeless session... 2024-03-29T10:40:22.825 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.842+0000 7f3908f10640 10 client.4265.objecter successfully canceled tick 2024-03-29T10:40:22.826 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.842+0000 7f3908f10640 1 -- 172.21.15.57:0/4221829994 >> v1:172.21.15.179:6816/2698295936 conn(0x7f38e806dcc0 legacy=0x7f38e8070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:22.826 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.842+0000 7f3908f10640 10 monclient: shutdown 2024-03-29T10:40:22.826 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.842+0000 7f3908f10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:22.826 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.842+0000 7f3908f10640 1 -- 172.21.15.57:0/4221829994 >> v1:172.21.15.57:6790/0 conn(0x7f38fc038c30 legacy=0x7f38fc0bc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:22.826 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.842+0000 7f3908f10640 1 -- 172.21.15.57:0/4221829994 shutdown_connections 2024-03-29T10:40:22.826 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.842+0000 7f3908f10640 1 -- 172.21.15.57:0/4221829994 >> 172.21.15.57:0/4221829994 conn(0x7f38fc005bf0 msgr2=0x7f38fc0043a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:22.826 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.842+0000 7f3908f10640 1 -- 172.21.15.57:0/4221829994 shutdown_connections 2024-03-29T10:40:22.827 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.843+0000 7f3908f10640 1 -- 172.21.15.57:0/4221829994 wait complete. 2024-03-29T10:40:22.827 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:22.844+0000 7f3908f10640 1 librados: shutdown 2024-03-29T10:40:22.849 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:29.657 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:29.818 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.834+0000 7f52cc931640 10 monclient: get_monmap_and_config 2024-03-29T10:40:29.818 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.834+0000 7f52cc931640 10 monclient: build_initial_monmap 2024-03-29T10:40:29.818 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.834+0000 7f52cc931640 10 monclient: monmap: 2024-03-29T10:40:29.818 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:40:29.819 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:29.819 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:40:29.835644+0000 2024-03-29T10:40:29.819 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:40:29.835644+0000 2024-03-29T10:40:29.819 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:29.819 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:29.819 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:40:29.819 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:40:29.819 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:40:29.819 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:29.820 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.836+0000 7f52cc931640 1 Processor -- start 2024-03-29T10:40:29.820 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.836+0000 7f52cc931640 1 -- start start 2024-03-29T10:40:29.820 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.836+0000 7f52cc931640 10 monclient: init 2024-03-29T10:40:29.821 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.837+0000 7f52cc931640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:29.821 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.837+0000 7f52cc931640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:40:29.821 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.837+0000 7f52cc931640 10 monclient(hunting): picked mon.noname-a con 0x7f52c0036a60 addr v1:172.21.15.57:6789/0 2024-03-29T10:40:29.824 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.838+0000 7f52cc931640 10 monclient(hunting): picked mon.noname-c con 0x7f52c00c5de0 addr v1:172.21.15.179:6789/0 2024-03-29T10:40:29.825 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.842+0000 7f52c659c640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f52c00c5de0 0x7f52c00c2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:47347/0 (socket says 172.21.15.57:47347) 2024-03-29T10:40:29.825 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.842+0000 7f52cc931640 10 monclient(hunting): picked mon.noname-b con 0x7f52c00c3070 addr v1:172.21.15.57:6790/0 2024-03-29T10:40:29.826 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.842+0000 7f52cc931640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c002cc60 con 0x7f52c0036a60 2024-03-29T10:40:29.826 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.842+0000 7f52cc931640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c002e040 con 0x7f52c00c5de0 2024-03-29T10:40:29.826 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.842+0000 7f52cc931640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c002e320 con 0x7f52c00c3070 2024-03-29T10:40:29.826 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.842+0000 7f52cc931640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:29.826 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.842+0000 7f52cc931640 10 monclient(hunting): authenticate will time out at 982.018155s 2024-03-29T10:40:29.826 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.842+0000 7f52c659c640 1 -- 172.21.15.57:0/4173581845 learned_addr learned my addr 172.21.15.57:0/4173581845 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:40:29.830 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.844+0000 7f52c5d9b640 1 -- 172.21.15.57:0/4173581845 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437928390 0 0) 0x7f52c002e040 con 0x7f52c00c5de0 2024-03-29T10:40:29.830 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.844+0000 7f52c5d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:40:29.830 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.844+0000 7f52c5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:29.830 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.844+0000 7f52c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:29.830 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.844+0000 7f52c5d9b640 10 monclient(hunting): my global_id is 4255 2024-03-29T10:40:29.830 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.844+0000 7f52c5d9b640 1 -- 172.21.15.57:0/4173581845 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52b0003520 con 0x7f52c00c5de0 2024-03-29T10:40:29.830 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.844+0000 7f52c5d9b640 1 -- 172.21.15.57:0/4173581845 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286977415 0 0) 0x7f52c002cc60 con 0x7f52c0036a60 2024-03-29T10:40:29.830 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.844+0000 7f52c5d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:40:29.830 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.844+0000 7f52c5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:29.830 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.844+0000 7f52c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:29.830 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.844+0000 7f52c5d9b640 10 monclient(hunting): my global_id is 4248 2024-03-29T10:40:29.830 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.844+0000 7f52c5d9b640 1 -- 172.21.15.57:0/4173581845 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52b00048a0 con 0x7f52c0036a60 2024-03-29T10:40:29.830 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.844+0000 7f52c5d9b640 1 -- 172.21.15.57:0/4173581845 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 682394870 0 0) 0x7f52b00048a0 con 0x7f52c0036a60 2024-03-29T10:40:29.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.845+0000 7f52c5d9b640 1 -- 172.21.15.57:0/4173581845 >> v1:172.21.15.57:6790/0 conn(0x7f52c00c3070 legacy=0x7f52c00d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:40:29.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.845+0000 7f52c5d9b640 1 -- 172.21.15.57:0/4173581845 >> v1:172.21.15.179:6789/0 conn(0x7f52c00c5de0 legacy=0x7f52c00c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:29.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.845+0000 7f52c5d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:40:29.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.845+0000 7f52c5d9b640 1 monclient: found mon.noname-a 2024-03-29T10:40:29.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.845+0000 7f52c5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:29.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.845+0000 7f52c5d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:40:29.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.845+0000 7f52c5d9b640 1 -- 172.21.15.57:0/4173581845 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c002c830 con 0x7f52c0036a60 2024-03-29T10:40:29.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.845+0000 7f52c5d9b640 10 monclient: _finish_auth 0 2024-03-29T10:40:29.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.845+0000 7f52c5d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:40:29.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.845+0000 7f52c5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:29.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.845+0000 7f52c5d9b640 1 -- 172.21.15.57:0/4173581845 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f52b8001630 con 0x7f52c0036a60 2024-03-29T10:40:29.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.845+0000 7f52c5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:29.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.845+0000 7f52c5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:29.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.845+0000 7f52c5d9b640 10 monclient: dump: 2024-03-29T10:40:29.832 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:29.832 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:29.832 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:29.832 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:29.832 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:29.832 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:29.832 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:29.832 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:29.832 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:29.832 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:29.832 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.845+0000 7f52cc931640 5 monclient: authenticate success, global_id 4248 2024-03-29T10:40:29.832 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.845+0000 7f52cc931640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:29.832 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.845+0000 7f52c5d9b640 1 -- 172.21.15.57:0/4173581845 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52b8001880 con 0x7f52c0036a60 2024-03-29T10:40:29.832 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.845+0000 7f52c5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:29.833 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.845+0000 7f52c5d9b640 1 -- 172.21.15.57:0/4173581845 <== mon.0 v1:172.21.15.57:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f52b8001c10 con 0x7f52c0036a60 2024-03-29T10:40:29.833 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.845+0000 7f52c5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:29.833 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.845+0000 7f52c5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:29.833 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.845+0000 7f52c5d9b640 10 monclient: dump: 2024-03-29T10:40:29.833 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:29.833 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:29.833 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:29.833 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:29.833 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:29.833 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:29.833 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:29.834 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:29.834 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:29.834 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:29.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.845+0000 7f52cc931640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:29.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.845+0000 7f52cc931640 4 set_mon_vals no callback set 2024-03-29T10:40:29.835 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.846+0000 7f52cc931640 10 monclient: shutdown 2024-03-29T10:40:29.835 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.846+0000 7f52cc931640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:29.835 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.846+0000 7f52cc931640 1 -- 172.21.15.57:0/4173581845 >> v1:172.21.15.57:6789/0 conn(0x7f52c0036a60 legacy=0x7f52c00c56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:29.836 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.847+0000 7f52cc931640 1 -- 172.21.15.57:0/4173581845 shutdown_connections 2024-03-29T10:40:29.836 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.847+0000 7f52cc931640 1 -- 172.21.15.57:0/4173581845 >> 172.21.15.57:0/4173581845 conn(0x7f52c0005c00 msgr2=0x7f52c0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:29.836 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.848+0000 7f52cc931640 1 -- 172.21.15.57:0/4173581845 shutdown_connections 2024-03-29T10:40:29.836 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.849+0000 7f52cc931640 1 -- 172.21.15.57:0/4173581845 wait complete. 2024-03-29T10:40:29.836 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.850+0000 7f52cc931640 10 monclient: build_initial_monmap 2024-03-29T10:40:29.836 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.850+0000 7f52cc931640 10 monclient: monmap: 2024-03-29T10:40:29.836 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:40:29.836 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:29.836 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:40:29.836 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:40:29.836 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:29.836 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:29.836 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:40:29.837 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:40:29.837 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:40:29.837 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:29.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.851+0000 7f52cc931640 1 librados: starting msgr at 2024-03-29T10:40:29.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.851+0000 7f52cc931640 1 librados: starting objecter 2024-03-29T10:40:29.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.851+0000 7f52cc931640 1 Processor -- start 2024-03-29T10:40:29.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.851+0000 7f52cc931640 1 -- start start 2024-03-29T10:40:29.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.851+0000 7f52cc931640 1 librados: setting wanted keys 2024-03-29T10:40:29.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.851+0000 7f52cc931640 1 librados: calling monclient init 2024-03-29T10:40:29.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.851+0000 7f52cc931640 10 monclient: init 2024-03-29T10:40:29.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.852+0000 7f52cc931640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:29.839 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.852+0000 7f52cc931640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:29.839 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.853+0000 7f52cc931640 10 monclient(hunting): picked mon.noname-a con 0x7f52c0036a60 addr v1:172.21.15.57:6789/0 2024-03-29T10:40:29.839 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.853+0000 7f52cc931640 10 monclient(hunting): picked mon.noname-c con 0x7f52c00c3070 addr v1:172.21.15.57:6790/0 2024-03-29T10:40:29.839 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.854+0000 7f52cc931640 10 monclient(hunting): picked mon.noname-b con 0x7f52c002fa60 addr v1:172.21.15.179:6789/0 2024-03-29T10:40:29.839 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.854+0000 7f52cc931640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c0031390 con 0x7f52c0036a60 2024-03-29T10:40:29.839 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.854+0000 7f52cc931640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c0030040 con 0x7f52c002fa60 2024-03-29T10:40:29.839 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.855+0000 7f52cc931640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c0033bb0 con 0x7f52c00c3070 2024-03-29T10:40:29.839 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.855+0000 7f52cc931640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:29.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.855+0000 7f52cc931640 10 monclient(hunting): authenticate will time out at 982.031038s 2024-03-29T10:40:29.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.857+0000 7f52c6d9d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f52c0036a60 0x7f52c002ed70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:55565/0 (socket says 172.21.15.57:55565) 2024-03-29T10:40:29.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.857+0000 7f52c6d9d640 1 -- 172.21.15.57:0/1452010207 learned_addr learned my addr 172.21.15.57:0/1452010207 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:40:29.843 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.857+0000 7f52af7fe640 10 client.?.objecter ms_handle_connect 0x7f52c0036a60 2024-03-29T10:40:29.843 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.857+0000 7f52af7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:29.843 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.860+0000 7f52af7fe640 10 client.?.objecter ms_handle_connect 0x7f52c00c3070 2024-03-29T10:40:29.843 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.860+0000 7f52af7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:29.843 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.860+0000 7f52af7fe640 1 -- 172.21.15.57:0/1452010207 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263365927 0 0) 0x7f52c0033bb0 con 0x7f52c00c3070 2024-03-29T10:40:29.843 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.860+0000 7f52af7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:29.843 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.860+0000 7f52af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:29.844 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.860+0000 7f52af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:29.844 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.860+0000 7f52af7fe640 10 monclient(hunting): my global_id is 4268 2024-03-29T10:40:29.844 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.860+0000 7f52af7fe640 1 -- 172.21.15.57:0/1452010207 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52b4004eb0 con 0x7f52c00c3070 2024-03-29T10:40:29.844 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.860+0000 7f52af7fe640 1 -- 172.21.15.57:0/1452010207 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3534544381 0 0) 0x7f52c0031390 con 0x7f52c0036a60 2024-03-29T10:40:29.844 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.860+0000 7f52af7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:29.844 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.860+0000 7f52af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:29.844 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.860+0000 7f52af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:29.844 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.860+0000 7f52af7fe640 10 monclient(hunting): my global_id is 4251 2024-03-29T10:40:29.844 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.860+0000 7f52af7fe640 1 -- 172.21.15.57:0/1452010207 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52c0033bb0 con 0x7f52c0036a60 2024-03-29T10:40:29.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.861+0000 7f52af7fe640 1 -- 172.21.15.57:0/1452010207 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2172867451 0 0) 0x7f52c0033bb0 con 0x7f52c0036a60 2024-03-29T10:40:29.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.861+0000 7f52af7fe640 1 -- 172.21.15.57:0/1452010207 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52c0031390 con 0x7f52c0036a60 2024-03-29T10:40:29.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.861+0000 7f52af7fe640 1 -- 172.21.15.57:0/1452010207 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f52b8001c50 con 0x7f52c0036a60 2024-03-29T10:40:29.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.861+0000 7f52af7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:29.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.861+0000 7f52af7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:29.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.861+0000 7f52af7fe640 10 monclient(hunting): dump: 2024-03-29T10:40:29.852 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:29.852 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:29.852 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:29.852 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:29.853 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:29.853 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:29.853 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:29.853 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:29.853 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:29.853 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:29.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.861+0000 7f52af7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:29.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.861+0000 7f52af7fe640 1 -- 172.21.15.57:0/1452010207 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2400769195 0 0) 0x7f52b4004eb0 con 0x7f52c00c3070 2024-03-29T10:40:29.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.861+0000 7f52af7fe640 1 -- 172.21.15.57:0/1452010207 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52c0033bb0 con 0x7f52c00c3070 2024-03-29T10:40:29.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.861+0000 7f52af7fe640 1 -- 172.21.15.57:0/1452010207 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f52bc001830 con 0x7f52c00c3070 2024-03-29T10:40:29.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.861+0000 7f52af7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:29.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.861+0000 7f52af7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:29.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.861+0000 7f52af7fe640 10 monclient(hunting): dump: 2024-03-29T10:40:29.853 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:29.854 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:29.854 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:29.854 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:29.854 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:29.854 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:29.854 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:29.854 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:29.854 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:29.854 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:29.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.861+0000 7f52af7fe640 1 -- 172.21.15.57:0/1452010207 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1111044366 0 0) 0x7f52c0031390 con 0x7f52c0036a60 2024-03-29T10:40:29.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.861+0000 7f52af7fe640 1 -- 172.21.15.57:0/1452010207 >> v1:172.21.15.57:6790/0 conn(0x7f52c00c3070 legacy=0x7f52c015a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:29.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.861+0000 7f52af7fe640 1 -- 172.21.15.57:0/1452010207 >> v1:172.21.15.179:6789/0 conn(0x7f52c002fa60 legacy=0x7f52c015ae80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:40:29.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.861+0000 7f52af7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:40:29.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.861+0000 7f52af7fe640 1 monclient: found mon.a 2024-03-29T10:40:29.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.861+0000 7f52af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:29.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.861+0000 7f52af7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:40:29.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.861+0000 7f52af7fe640 1 -- 172.21.15.57:0/1452010207 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c0033d40 con 0x7f52c0036a60 2024-03-29T10:40:29.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.861+0000 7f52af7fe640 10 monclient: _finish_auth 0 2024-03-29T10:40:29.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.861+0000 7f52af7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:40:29.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.861+0000 7f52af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:29.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.862+0000 7f52cc931640 5 monclient: authenticate success, global_id 4251 2024-03-29T10:40:29.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.862+0000 7f52cc931640 10 monclient: _renew_subs 2024-03-29T10:40:29.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.862+0000 7f52cc931640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:40:29.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.862+0000 7f52cc931640 1 -- 172.21.15.57:0/1452010207 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c0033f90 con 0x7f52c0036a60 2024-03-29T10:40:29.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.862+0000 7f52cc931640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:29.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.862+0000 7f52cc931640 10 monclient: _renew_subs 2024-03-29T10:40:29.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.862+0000 7f52cc931640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:40:29.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.862+0000 7f52cc931640 1 -- 172.21.15.57:0/1452010207 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c00344c0 con 0x7f52c0036a60 2024-03-29T10:40:29.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.862+0000 7f52cc931640 1 librados: init done 2024-03-29T10:40:29.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.862+0000 7f52af7fe640 1 -- 172.21.15.57:0/1452010207 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52b8003660 con 0x7f52c0036a60 2024-03-29T10:40:29.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.862+0000 7f52af7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:29.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.862+0000 7f52c559a640 4 set_mon_vals no callback set 2024-03-29T10:40:29.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.863+0000 7f52af7fe640 1 -- 172.21.15.57:0/1452010207 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f52b8003c40 con 0x7f52c0036a60 2024-03-29T10:40:29.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.864+0000 7f52af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:29.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.863+0000 7f52cc931640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:29.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.864+0000 7f52af7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:29.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.864+0000 7f52af7fe640 10 monclient: dump: 2024-03-29T10:40:29.856 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:29.857 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:29.857 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:29.857 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:29.857 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:29.857 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:29.857 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:29.857 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:29.857 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:29.857 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:29.857 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.864+0000 7f52af7fe640 1 -- 172.21.15.57:0/1452010207 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f52b8003ea0 con 0x7f52c0036a60 2024-03-29T10:40:29.857 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.864+0000 7f52cc931640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:29.857 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.865+0000 7f52cc931640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:40:29.857 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.865+0000 7f52cc931640 1 -- 172.21.15.57:0/1452010207 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a8005080 con 0x7f52c0036a60 2024-03-29T10:40:29.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.865+0000 7f52af7fe640 1 -- 172.21.15.57:0/1452010207 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f52b8084c90 con 0x7f52c0036a60 2024-03-29T10:40:29.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.866+0000 7f52af7fe640 10 client.4251.objecter ms_dispatch 0x7f52c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:29.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.866+0000 7f52af7fe640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:29.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.866+0000 7f52c659c640 10 monclient: get_auth_request con 0x7f52b406dd90 auth_method 0 2024-03-29T10:40:29.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.866+0000 7f52af7fe640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:29.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.866+0000 7f52af7fe640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-29T10:40:29.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.867+0000 7f52af7fe640 10 client.4251.objecter ms_handle_connect 0x7f52b406dd90 2024-03-29T10:40:29.859 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.875+0000 7f52af7fe640 1 -- 172.21.15.57:0/1452010207 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f52b8050570 con 0x7f52c0036a60 2024-03-29T10:40:29.859 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.875+0000 7f52af7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:29.859 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:29.875+0000 7f52af7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:30.191 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:30.208+0000 7f52cc931640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:30.192 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:30.208+0000 7f52cc931640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:30.192 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:30.208+0000 7f52cc931640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:40:30.192 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:30.208+0000 7f52cc931640 1 -- 172.21.15.57:0/1452010207 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52a8005330 con 0x7f52c0036a60 2024-03-29T10:40:30.192 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:30.209+0000 7f52af7fe640 1 -- 172.21.15.57:0/1452010207 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f52b80524a0 con 0x7f52c0036a60 2024-03-29T10:40:30.192 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:30.209+0000 7f52af7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:30.192 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:30.209+0000 7f52af7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:30.193 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:40:30.193 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:30.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:30.212+0000 7f52cc931640 10 librados: watch_flush enter 2024-03-29T10:40:30.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:30.214+0000 7f52cc931640 10 librados: watch_flush exit 2024-03-29T10:40:30.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:30.214+0000 7f52cc931640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-29T10:40:30.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:30.214+0000 7f52cc931640 10 client.4251.objecter successfully canceled tick 2024-03-29T10:40:30.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:30.214+0000 7f52cc931640 1 -- 172.21.15.57:0/1452010207 >> v1:172.21.15.179:6816/2698295936 conn(0x7f52b406dd90 legacy=0x7f52b4070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:30.199 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:30.214+0000 7f52cc931640 10 monclient: shutdown 2024-03-29T10:40:30.199 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:30.214+0000 7f52cc931640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:30.199 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:30.214+0000 7f52cc931640 1 -- 172.21.15.57:0/1452010207 >> v1:172.21.15.57:6789/0 conn(0x7f52c0036a60 legacy=0x7f52c002ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:30.199 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:30.215+0000 7f52cc931640 1 -- 172.21.15.57:0/1452010207 shutdown_connections 2024-03-29T10:40:30.199 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:30.215+0000 7f52cc931640 1 -- 172.21.15.57:0/1452010207 >> 172.21.15.57:0/1452010207 conn(0x7f52c0005c00 msgr2=0x7f52c00c2730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:30.199 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:30.215+0000 7f52cc931640 1 -- 172.21.15.57:0/1452010207 shutdown_connections 2024-03-29T10:40:30.199 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:30.215+0000 7f52cc931640 1 -- 172.21.15.57:0/1452010207 wait complete. 2024-03-29T10:40:30.199 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:30.215+0000 7f52cc931640 1 librados: shutdown 2024-03-29T10:40:30.216 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:37.023 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:37.171 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.187+0000 7fb79152e640 10 monclient: get_monmap_and_config 2024-03-29T10:40:37.172 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.187+0000 7fb79152e640 10 monclient: build_initial_monmap 2024-03-29T10:40:37.172 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.187+0000 7fb79152e640 10 monclient: monmap: 2024-03-29T10:40:37.172 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:40:37.172 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:37.172 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:40:37.188126+0000 2024-03-29T10:40:37.172 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:40:37.188126+0000 2024-03-29T10:40:37.172 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:37.172 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:37.172 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:40:37.172 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:40:37.172 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:40:37.172 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:37.173 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.188+0000 7fb79152e640 1 Processor -- start 2024-03-29T10:40:37.173 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.188+0000 7fb79152e640 1 -- start start 2024-03-29T10:40:37.173 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.188+0000 7fb79152e640 10 monclient: init 2024-03-29T10:40:37.176 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.189+0000 7fb79152e640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:37.176 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.189+0000 7fb79152e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:37.176 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.189+0000 7fb79152e640 10 monclient(hunting): picked mon.noname-a con 0x7fb78c0ae490 addr v1:172.21.15.57:6789/0 2024-03-29T10:40:37.176 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.189+0000 7fb79152e640 10 monclient(hunting): picked mon.noname-b con 0x7fb78c0aa520 addr v1:172.21.15.57:6790/0 2024-03-29T10:40:37.176 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.189+0000 7fb79152e640 10 monclient(hunting): picked mon.noname-c con 0x7fb78c0ad160 addr v1:172.21.15.179:6789/0 2024-03-29T10:40:37.176 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.189+0000 7fb79152e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb78c0aac70 con 0x7fb78c0ae490 2024-03-29T10:40:37.176 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.189+0000 7fb79152e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb78c0ac630 con 0x7fb78c0ad160 2024-03-29T10:40:37.176 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.189+0000 7fb79152e640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb78c0aaf10 con 0x7fb78c0aa520 2024-03-29T10:40:37.176 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.189+0000 7fb79152e640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:37.176 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.189+0000 7fb79152e640 10 monclient(hunting): authenticate will time out at 989.365544s 2024-03-29T10:40:37.178 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.195+0000 7fb78b7fe640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7fb78c0aa520 0x7fb78c0aca50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:46807/0 (socket says 172.21.15.57:46807) 2024-03-29T10:40:37.179 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.195+0000 7fb78b7fe640 1 -- 172.21.15.57:0/1092819360 learned_addr learned my addr 172.21.15.57:0/1092819360 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:40:37.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 1 -- 172.21.15.57:0/1092819360 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424561130 0 0) 0x7fb78c0aaf10 con 0x7fb78c0aa520 2024-03-29T10:40:37.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:37.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:37.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:37.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 10 monclient(hunting): my global_id is 4271 2024-03-29T10:40:37.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 1 -- 172.21.15.57:0/1092819360 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb774003520 con 0x7fb78c0aa520 2024-03-29T10:40:37.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 1 -- 172.21.15.57:0/1092819360 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181897285 0 0) 0x7fb78c0aac70 con 0x7fb78c0ae490 2024-03-29T10:40:37.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:37.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:37.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:37.181 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 10 monclient(hunting): my global_id is 4254 2024-03-29T10:40:37.181 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 1 -- 172.21.15.57:0/1092819360 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb78c0aaf10 con 0x7fb78c0ae490 2024-03-29T10:40:37.181 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 1 -- 172.21.15.57:0/1092819360 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2147583658 0 0) 0x7fb774003520 con 0x7fb78c0aa520 2024-03-29T10:40:37.181 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 1 -- 172.21.15.57:0/1092819360 >> v1:172.21.15.179:6789/0 conn(0x7fb78c0ad160 legacy=0x7fb78c151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:40:37.181 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 1 -- 172.21.15.57:0/1092819360 >> v1:172.21.15.57:6789/0 conn(0x7fb78c0ae490 legacy=0x7fb78c143680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:37.181 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 10 monclient: _finish_hunting 0 2024-03-29T10:40:37.181 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 1 monclient: found mon.noname-b 2024-03-29T10:40:37.181 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:37.181 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:40:37.181 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 1 -- 172.21.15.57:0/1092819360 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c0aded0 con 0x7fb78c0aa520 2024-03-29T10:40:37.181 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 10 monclient: _finish_auth 0 2024-03-29T10:40:37.181 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 10 monclient: _check_auth_tickets 2024-03-29T10:40:37.181 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:37.181 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb79152e640 5 monclient: authenticate success, global_id 4271 2024-03-29T10:40:37.182 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb79152e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:37.182 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 1 -- 172.21.15.57:0/1092819360 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb784001a10 con 0x7fb78c0aa520 2024-03-29T10:40:37.182 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:37.182 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:37.182 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 10 monclient: dump: 2024-03-29T10:40:37.182 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:37.182 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:37.182 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:37.182 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:37.182 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:37.182 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:37.182 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:37.182 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:37.182 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:37.182 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:37.183 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb79152e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:37.183 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 1 -- 172.21.15.57:0/1092819360 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb784001c30 con 0x7fb78c0aa520 2024-03-29T10:40:37.183 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:37.183 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb79152e640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:37.183 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb79152e640 4 set_mon_vals no callback set 2024-03-29T10:40:37.183 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 1 -- 172.21.15.57:0/1092819360 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb784001fc0 con 0x7fb78c0aa520 2024-03-29T10:40:37.183 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:37.183 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:37.183 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.196+0000 7fb78affd640 10 monclient: dump: 2024-03-29T10:40:37.183 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:37.183 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:37.183 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:37.183 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:37.183 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:37.183 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:37.184 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:37.184 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:37.184 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:37.184 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:37.184 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.197+0000 7fb79152e640 10 monclient: shutdown 2024-03-29T10:40:37.184 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.197+0000 7fb79152e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:37.184 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.197+0000 7fb79152e640 1 -- 172.21.15.57:0/1092819360 >> v1:172.21.15.57:6790/0 conn(0x7fb78c0aa520 legacy=0x7fb78c0aca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:37.184 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.197+0000 7fb79152e640 1 -- 172.21.15.57:0/1092819360 shutdown_connections 2024-03-29T10:40:37.186 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.197+0000 7fb79152e640 1 -- 172.21.15.57:0/1092819360 >> 172.21.15.57:0/1092819360 conn(0x7fb78c0b3a30 msgr2=0x7fb78c0b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:37.186 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.197+0000 7fb79152e640 1 -- 172.21.15.57:0/1092819360 shutdown_connections 2024-03-29T10:40:37.186 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.197+0000 7fb79152e640 1 -- 172.21.15.57:0/1092819360 wait complete. 2024-03-29T10:40:37.187 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.198+0000 7fb79152e640 10 monclient: build_initial_monmap 2024-03-29T10:40:37.187 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.198+0000 7fb79152e640 10 monclient: monmap: 2024-03-29T10:40:37.187 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:40:37.187 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:37.187 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:40:37.187 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:40:37.187 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:37.187 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:37.187 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:40:37.187 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:40:37.187 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:40:37.187 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:37.187 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.199+0000 7fb79152e640 1 librados: starting msgr at 2024-03-29T10:40:37.187 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.199+0000 7fb79152e640 1 librados: starting objecter 2024-03-29T10:40:37.188 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.199+0000 7fb79152e640 1 Processor -- start 2024-03-29T10:40:37.188 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.199+0000 7fb79152e640 1 -- start start 2024-03-29T10:40:37.188 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.199+0000 7fb79152e640 1 librados: setting wanted keys 2024-03-29T10:40:37.188 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.199+0000 7fb79152e640 1 librados: calling monclient init 2024-03-29T10:40:37.188 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.199+0000 7fb79152e640 10 monclient: init 2024-03-29T10:40:37.188 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.200+0000 7fb79152e640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:37.188 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.200+0000 7fb79152e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:40:37.188 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.200+0000 7fb79152e640 10 monclient(hunting): picked mon.noname-c con 0x7fb78c0aa520 addr v1:172.21.15.57:6790/0 2024-03-29T10:40:37.188 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.200+0000 7fb79152e640 10 monclient(hunting): picked mon.noname-a con 0x7fb78c0ad160 addr v1:172.21.15.57:6789/0 2024-03-29T10:40:37.188 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.200+0000 7fb79152e640 10 monclient(hunting): picked mon.noname-b con 0x7fb78c0ae490 addr v1:172.21.15.179:6789/0 2024-03-29T10:40:37.188 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.200+0000 7fb79152e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb78c13c370 con 0x7fb78c0ad160 2024-03-29T10:40:37.188 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.200+0000 7fb79152e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb78c145fb0 con 0x7fb78c0ae490 2024-03-29T10:40:37.188 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.200+0000 7fb79152e640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb78c146140 con 0x7fb78c0aa520 2024-03-29T10:40:37.188 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.200+0000 7fb79152e640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:37.188 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.200+0000 7fb79152e640 10 monclient(hunting): authenticate will time out at 989.376412s 2024-03-29T10:40:37.189 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.205+0000 7fb78b7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fb78c0ad160 0x7fb78c134320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:48509/0 (socket says 172.21.15.57:48509) 2024-03-29T10:40:37.189 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.205+0000 7fb78b7fe640 1 -- 172.21.15.57:0/648779439 learned_addr learned my addr 172.21.15.57:0/648779439 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:40:37.191 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.206+0000 7fb788ff9640 10 client.?.objecter ms_handle_connect 0x7fb78c0ad160 2024-03-29T10:40:37.191 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.207+0000 7fb788ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:37.191 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.207+0000 7fb788ff9640 10 client.?.objecter ms_handle_connect 0x7fb78c0aa520 2024-03-29T10:40:37.191 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.207+0000 7fb788ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:37.191 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.207+0000 7fb788ff9640 1 -- 172.21.15.57:0/648779439 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265237441 0 0) 0x7fb78c146140 con 0x7fb78c0aa520 2024-03-29T10:40:37.191 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.207+0000 7fb788ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:40:37.191 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.207+0000 7fb788ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:37.191 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.207+0000 7fb788ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:37.191 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.207+0000 7fb788ff9640 10 monclient(hunting): my global_id is 4274 2024-03-29T10:40:37.191 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.207+0000 7fb788ff9640 1 -- 172.21.15.57:0/648779439 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb778004eb0 con 0x7fb78c0aa520 2024-03-29T10:40:37.191 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.208+0000 7fb788ff9640 1 -- 172.21.15.57:0/648779439 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1577100052 0 0) 0x7fb778004eb0 con 0x7fb78c0aa520 2024-03-29T10:40:37.194 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.208+0000 7fb788ff9640 1 -- 172.21.15.57:0/648779439 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb78c146140 con 0x7fb78c0aa520 2024-03-29T10:40:37.194 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.208+0000 7fb788ff9640 1 -- 172.21.15.57:0/648779439 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb77c001930 con 0x7fb78c0aa520 2024-03-29T10:40:37.194 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.208+0000 7fb788ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:37.194 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.208+0000 7fb788ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:37.194 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.209+0000 7fb788ff9640 10 monclient(hunting): dump: 2024-03-29T10:40:37.194 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:37.194 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:37.194 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:37.194 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:37.194 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:37.194 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:37.195 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:37.195 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:37.195 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:37.195 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:37.195 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.209+0000 7fb788ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:37.195 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.209+0000 7fb788ff9640 10 client.?.objecter ms_handle_connect 0x7fb78c0ae490 2024-03-29T10:40:37.195 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.209+0000 7fb788ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:37.195 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.209+0000 7fb788ff9640 1 -- 172.21.15.57:0/648779439 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2575594809 0 0) 0x7fb78c146140 con 0x7fb78c0aa520 2024-03-29T10:40:37.195 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.209+0000 7fb788ff9640 1 -- 172.21.15.57:0/648779439 >> v1:172.21.15.179:6789/0 conn(0x7fb78c0ae490 legacy=0x7fb78c1de600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:37.195 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.209+0000 7fb788ff9640 1 -- 172.21.15.57:0/648779439 >> v1:172.21.15.57:6789/0 conn(0x7fb78c0ad160 legacy=0x7fb78c134320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:37.195 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.209+0000 7fb788ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:40:37.195 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.209+0000 7fb788ff9640 1 monclient: found mon.c 2024-03-29T10:40:37.195 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.209+0000 7fb788ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:37.195 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.209+0000 7fb788ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:40:37.195 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.210+0000 7fb788ff9640 1 -- 172.21.15.57:0/648779439 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c1424d0 con 0x7fb78c0aa520 2024-03-29T10:40:37.196 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.210+0000 7fb788ff9640 10 monclient: _finish_auth 0 2024-03-29T10:40:37.196 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.210+0000 7fb788ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:40:37.196 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.210+0000 7fb788ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:37.196 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.210+0000 7fb79152e640 5 monclient: authenticate success, global_id 4274 2024-03-29T10:40:37.196 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.210+0000 7fb79152e640 10 monclient: _renew_subs 2024-03-29T10:40:37.196 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.210+0000 7fb79152e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:40:37.196 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.210+0000 7fb79152e640 1 -- 172.21.15.57:0/648779439 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb78c142780 con 0x7fb78c0aa520 2024-03-29T10:40:37.196 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.210+0000 7fb79152e640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:37.196 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.210+0000 7fb79152e640 10 monclient: _renew_subs 2024-03-29T10:40:37.196 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.210+0000 7fb79152e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:40:37.196 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.210+0000 7fb79152e640 1 -- 172.21.15.57:0/648779439 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb78c134c10 con 0x7fb78c0aa520 2024-03-29T10:40:37.196 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.210+0000 7fb79152e640 1 librados: init done 2024-03-29T10:40:37.196 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.210+0000 7fb788ff9640 1 -- 172.21.15.57:0/648779439 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb77c003610 con 0x7fb78c0aa520 2024-03-29T10:40:37.196 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.210+0000 7fb788ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:37.196 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.210+0000 7fb788ff9640 1 -- 172.21.15.57:0/648779439 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb77c003920 con 0x7fb78c0aa520 2024-03-29T10:40:37.196 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.210+0000 7fb788ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:37.197 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.210+0000 7fb788ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:37.197 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.210+0000 7fb788ff9640 10 monclient: dump: 2024-03-29T10:40:37.197 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:37.197 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:37.197 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:37.197 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:37.197 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:37.197 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:37.197 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:37.197 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:37.197 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:37.197 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:37.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.211+0000 7fb79152e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:37.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.211+0000 7fb79152e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:37.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.211+0000 7fb79152e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:40:37.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.211+0000 7fb79152e640 1 -- 172.21.15.57:0/648779439 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb78c146720 con 0x7fb78c0aa520 2024-03-29T10:40:37.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.213+0000 7fb78a7fc640 4 set_mon_vals no callback set 2024-03-29T10:40:37.199 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.215+0000 7fb788ff9640 1 -- 172.21.15.57:0/648779439 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fb77c0198f0 con 0x7fb78c0aa520 2024-03-29T10:40:37.200 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.216+0000 7fb78b7fe640 10 monclient: get_auth_request con 0x7fb77806dbd0 auth_method 0 2024-03-29T10:40:37.200 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.216+0000 7fb788ff9640 1 -- 172.21.15.57:0/648779439 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7fb77c083910 con 0x7fb78c0aa520 2024-03-29T10:40:37.200 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.217+0000 7fb788ff9640 10 client.4274.objecter ms_dispatch 0x7fb78c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:37.200 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.217+0000 7fb788ff9640 3 client.4274.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:37.200 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.217+0000 7fb788ff9640 3 client.4274.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:37.200 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.217+0000 7fb788ff9640 20 client.4274.objecter dump_active .. 0 homeless 2024-03-29T10:40:37.200 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.217+0000 7fb788ff9640 10 client.4274.objecter ms_handle_connect 0x7fb77806dbd0 2024-03-29T10:40:37.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.222+0000 7fb788ff9640 1 -- 172.21.15.57:0/648779439 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb77c04f020 con 0x7fb78c0aa520 2024-03-29T10:40:37.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.222+0000 7fb788ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:37.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.222+0000 7fb788ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:37.551 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.568+0000 7fb79152e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:37.551 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.568+0000 7fb79152e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:37.551 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.568+0000 7fb79152e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:40:37.551 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.568+0000 7fb79152e640 1 -- 172.21.15.57:0/648779439 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb78c04f730 con 0x7fb78c0aa520 2024-03-29T10:40:37.553 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.570+0000 7fb788ff9640 1 -- 172.21.15.57:0/648779439 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7fb77c051120 con 0x7fb78c0aa520 2024-03-29T10:40:37.553 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.570+0000 7fb788ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:37.554 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.570+0000 7fb788ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:37.554 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:40:37.554 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:37.557 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.574+0000 7fb79152e640 10 librados: watch_flush enter 2024-03-29T10:40:37.557 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.574+0000 7fb79152e640 10 librados: watch_flush exit 2024-03-29T10:40:37.557 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.574+0000 7fb79152e640 20 client.4274.objecter shutdown clearing up homeless session... 2024-03-29T10:40:37.557 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.574+0000 7fb79152e640 10 client.4274.objecter successfully canceled tick 2024-03-29T10:40:37.559 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.575+0000 7fb79152e640 1 -- 172.21.15.57:0/648779439 >> v1:172.21.15.179:6816/2698295936 conn(0x7fb77806dbd0 legacy=0x7fb778070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:37.559 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.575+0000 7fb79152e640 10 monclient: shutdown 2024-03-29T10:40:37.559 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.575+0000 7fb79152e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:37.559 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.575+0000 7fb79152e640 1 -- 172.21.15.57:0/648779439 >> v1:172.21.15.57:6790/0 conn(0x7fb78c0aa520 legacy=0x7fb78c1458a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:37.559 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.576+0000 7fb79152e640 1 -- 172.21.15.57:0/648779439 shutdown_connections 2024-03-29T10:40:37.559 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.576+0000 7fb79152e640 1 -- 172.21.15.57:0/648779439 >> 172.21.15.57:0/648779439 conn(0x7fb78c0b3a30 msgr2=0x7fb78c0b1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:37.564 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.581+0000 7fb79152e640 1 -- 172.21.15.57:0/648779439 shutdown_connections 2024-03-29T10:40:37.566 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.583+0000 7fb79152e640 1 -- 172.21.15.57:0/648779439 wait complete. 2024-03-29T10:40:37.567 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:37.584+0000 7fb79152e640 1 librados: shutdown 2024-03-29T10:40:37.586 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:44.394 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:44.548 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.564+0000 7fcbe4aa8640 10 monclient: get_monmap_and_config 2024-03-29T10:40:44.548 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.564+0000 7fcbe4aa8640 10 monclient: build_initial_monmap 2024-03-29T10:40:44.548 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.564+0000 7fcbe4aa8640 10 monclient: monmap: 2024-03-29T10:40:44.548 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:40:44.548 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:44.548 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:40:44.565595+0000 2024-03-29T10:40:44.548 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:40:44.565595+0000 2024-03-29T10:40:44.548 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:44.549 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:44.549 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:40:44.549 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:40:44.549 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:40:44.549 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:44.549 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.565+0000 7fcbe4aa8640 1 Processor -- start 2024-03-29T10:40:44.551 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.568+0000 7fcbe4aa8640 1 -- start start 2024-03-29T10:40:44.551 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.568+0000 7fcbe4aa8640 10 monclient: init 2024-03-29T10:40:44.553 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.569+0000 7fcbe4aa8640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:44.553 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.569+0000 7fcbe4aa8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:40:44.553 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.569+0000 7fcbe4aa8640 10 monclient(hunting): picked mon.noname-a con 0x7fcbe006ee40 addr v1:172.21.15.57:6789/0 2024-03-29T10:40:44.553 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.569+0000 7fcbe4aa8640 10 monclient(hunting): picked mon.noname-c con 0x7fcbe00aeda0 addr v1:172.21.15.179:6789/0 2024-03-29T10:40:44.553 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.569+0000 7fcbe4aa8640 10 monclient(hunting): picked mon.noname-b con 0x7fcbe013c470 addr v1:172.21.15.57:6790/0 2024-03-29T10:40:44.553 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.569+0000 7fcbe4aa8640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe006fdb0 con 0x7fcbe006ee40 2024-03-29T10:40:44.553 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.569+0000 7fcbe4aa8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe0071090 con 0x7fcbe00aeda0 2024-03-29T10:40:44.553 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.569+0000 7fcbe4aa8640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe0071370 con 0x7fcbe013c470 2024-03-29T10:40:44.553 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.569+0000 7fcbe4aa8640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:44.553 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.569+0000 7fcbe4aa8640 10 monclient(hunting): authenticate will time out at 996.745333s 2024-03-29T10:40:44.556 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.573+0000 7fcbded9d640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7fcbe013c470 0x7fcbe014a640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:42893/0 (socket says 172.21.15.57:42893) 2024-03-29T10:40:44.556 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.573+0000 7fcbded9d640 1 -- 172.21.15.57:0/3330896598 learned_addr learned my addr 172.21.15.57:0/3330896598 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:40:44.560 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.576+0000 7fcbdd59a640 1 -- 172.21.15.57:0/3330896598 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394338204 0 0) 0x7fcbe0071370 con 0x7fcbe013c470 2024-03-29T10:40:44.560 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.576+0000 7fcbdd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:40:44.560 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.576+0000 7fcbdd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:44.560 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.576+0000 7fcbdd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:44.560 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.576+0000 7fcbdd59a640 10 monclient(hunting): my global_id is 4277 2024-03-29T10:40:44.560 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.576+0000 7fcbdd59a640 1 -- 172.21.15.57:0/3330896598 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbc8003520 con 0x7fcbe013c470 2024-03-29T10:40:44.560 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.576+0000 7fcbdd59a640 1 -- 172.21.15.57:0/3330896598 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851353775 0 0) 0x7fcbe006fdb0 con 0x7fcbe006ee40 2024-03-29T10:40:44.561 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.576+0000 7fcbdd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:40:44.561 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.576+0000 7fcbdd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:44.561 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.576+0000 7fcbdd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:44.561 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.576+0000 7fcbdd59a640 10 monclient(hunting): my global_id is 4257 2024-03-29T10:40:44.561 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.576+0000 7fcbdd59a640 1 -- 172.21.15.57:0/3330896598 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbe0071370 con 0x7fcbe006ee40 2024-03-29T10:40:44.561 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.577+0000 7fcbdd59a640 1 -- 172.21.15.57:0/3330896598 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 197274707 0 0) 0x7fcbe0071370 con 0x7fcbe006ee40 2024-03-29T10:40:44.561 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.577+0000 7fcbdd59a640 1 -- 172.21.15.57:0/3330896598 >> v1:172.21.15.57:6790/0 conn(0x7fcbe013c470 legacy=0x7fcbe014a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:44.561 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.577+0000 7fcbdd59a640 1 -- 172.21.15.57:0/3330896598 >> v1:172.21.15.179:6789/0 conn(0x7fcbe00aeda0 legacy=0x7fcbe0071790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:44.561 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.577+0000 7fcbdd59a640 10 monclient: _finish_hunting 0 2024-03-29T10:40:44.561 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.577+0000 7fcbdd59a640 1 monclient: found mon.noname-a 2024-03-29T10:40:44.562 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.577+0000 7fcbdd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:44.562 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.577+0000 7fcbdd59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:40:44.562 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.577+0000 7fcbdd59a640 1 -- 172.21.15.57:0/3330896598 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe0070220 con 0x7fcbe006ee40 2024-03-29T10:40:44.562 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.577+0000 7fcbdd59a640 10 monclient: _finish_auth 0 2024-03-29T10:40:44.562 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.577+0000 7fcbdd59a640 10 monclient: _check_auth_tickets 2024-03-29T10:40:44.562 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.577+0000 7fcbdd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:44.562 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.577+0000 7fcbe4aa8640 5 monclient: authenticate success, global_id 4257 2024-03-29T10:40:44.562 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.577+0000 7fcbe4aa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:44.562 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.577+0000 7fcbdd59a640 1 -- 172.21.15.57:0/3330896598 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fcbd80019f0 con 0x7fcbe006ee40 2024-03-29T10:40:44.562 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.577+0000 7fcbdd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:44.562 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.577+0000 7fcbdd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:44.562 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.577+0000 7fcbdd59a640 10 monclient: dump: 2024-03-29T10:40:44.563 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:44.563 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:44.563 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:44.563 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:44.563 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:44.563 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:44.563 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:44.563 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:44.563 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:44.563 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:44.563 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.578+0000 7fcbe4aa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:44.565 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.578+0000 7fcbdd59a640 1 -- 172.21.15.57:0/3330896598 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbd8001c10 con 0x7fcbe006ee40 2024-03-29T10:40:44.565 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.578+0000 7fcbdd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:44.565 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.578+0000 7fcbe4aa8640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:44.565 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.578+0000 7fcbe4aa8640 4 set_mon_vals no callback set 2024-03-29T10:40:44.565 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.580+0000 7fcbe4aa8640 10 monclient: shutdown 2024-03-29T10:40:44.565 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.580+0000 7fcbe4aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:44.565 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.580+0000 7fcbe4aa8640 1 -- 172.21.15.57:0/3330896598 >> v1:172.21.15.57:6789/0 conn(0x7fcbe006ee40 legacy=0x7fcbe013bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:44.565 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.580+0000 7fcbe4aa8640 1 -- 172.21.15.57:0/3330896598 shutdown_connections 2024-03-29T10:40:44.565 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.580+0000 7fcbe4aa8640 1 -- 172.21.15.57:0/3330896598 >> 172.21.15.57:0/3330896598 conn(0x7fcbe0067470 msgr2=0x7fcbe00b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:44.566 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.580+0000 7fcbe4aa8640 1 -- 172.21.15.57:0/3330896598 shutdown_connections 2024-03-29T10:40:44.566 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.581+0000 7fcbe4aa8640 1 -- 172.21.15.57:0/3330896598 wait complete. 2024-03-29T10:40:44.567 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.582+0000 7fcbe4aa8640 10 monclient: build_initial_monmap 2024-03-29T10:40:44.567 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.582+0000 7fcbe4aa8640 10 monclient: monmap: 2024-03-29T10:40:44.567 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:40:44.567 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:44.567 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:40:44.567 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:40:44.567 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:44.567 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:44.567 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:40:44.567 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:40:44.568 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:40:44.568 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:44.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.582+0000 7fcbe4aa8640 1 librados: starting msgr at 2024-03-29T10:40:44.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.582+0000 7fcbe4aa8640 1 librados: starting objecter 2024-03-29T10:40:44.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.582+0000 7fcbe4aa8640 1 Processor -- start 2024-03-29T10:40:44.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.582+0000 7fcbe4aa8640 1 -- start start 2024-03-29T10:40:44.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.582+0000 7fcbe4aa8640 1 librados: setting wanted keys 2024-03-29T10:40:44.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.582+0000 7fcbe4aa8640 1 librados: calling monclient init 2024-03-29T10:40:44.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.582+0000 7fcbe4aa8640 10 monclient: init 2024-03-29T10:40:44.569 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.583+0000 7fcbe4aa8640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:44.569 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.583+0000 7fcbe4aa8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:44.569 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.583+0000 7fcbe4aa8640 10 monclient(hunting): picked mon.noname-b con 0x7fcbe006ee40 addr v1:172.21.15.179:6789/0 2024-03-29T10:40:44.569 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.583+0000 7fcbe4aa8640 10 monclient(hunting): picked mon.noname-a con 0x7fcbe00aeda0 addr v1:172.21.15.57:6789/0 2024-03-29T10:40:44.569 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.583+0000 7fcbe4aa8640 10 monclient(hunting): picked mon.noname-c con 0x7fcbe013c470 addr v1:172.21.15.57:6790/0 2024-03-29T10:40:44.569 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.583+0000 7fcbe4aa8640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe01395b0 con 0x7fcbe00aeda0 2024-03-29T10:40:44.569 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.583+0000 7fcbe4aa8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe013e950 con 0x7fcbe006ee40 2024-03-29T10:40:44.569 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.583+0000 7fcbe4aa8640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe01df060 con 0x7fcbe013c470 2024-03-29T10:40:44.569 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.583+0000 7fcbe4aa8640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:44.569 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.583+0000 7fcbe4aa8640 10 monclient(hunting): authenticate will time out at 996.759860s 2024-03-29T10:40:44.578 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.590+0000 7fcbddd9b640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fcbe00aeda0 0x7fcbe0138510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:52623/0 (socket says 172.21.15.57:52623) 2024-03-29T10:40:44.578 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.590+0000 7fcbddd9b640 1 -- 172.21.15.57:0/3476602997 learned_addr learned my addr 172.21.15.57:0/3476602997 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:40:44.578 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.591+0000 7fcbceffd640 10 client.?.objecter ms_handle_connect 0x7fcbe00aeda0 2024-03-29T10:40:44.578 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.591+0000 7fcbceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:44.578 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.591+0000 7fcbceffd640 1 -- 172.21.15.57:0/3476602997 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620877385 0 0) 0x7fcbe0071090 con 0x7fcbe00aeda0 2024-03-29T10:40:44.578 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.591+0000 7fcbceffd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:44.578 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.591+0000 7fcbceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:44.578 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.591+0000 7fcbceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:44.578 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.591+0000 7fcbceffd640 10 monclient(hunting): my global_id is 4260 2024-03-29T10:40:44.579 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.591+0000 7fcbceffd640 1 -- 172.21.15.57:0/3476602997 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbc0004eb0 con 0x7fcbe00aeda0 2024-03-29T10:40:44.579 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.591+0000 7fcbceffd640 10 client.?.objecter ms_handle_connect 0x7fcbe006ee40 2024-03-29T10:40:44.582 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.591+0000 7fcbceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:44.582 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.592+0000 7fcbceffd640 1 -- 172.21.15.57:0/3476602997 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 224498062 0 0) 0x7fcbc0004eb0 con 0x7fcbe00aeda0 2024-03-29T10:40:44.582 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.592+0000 7fcbceffd640 1 -- 172.21.15.57:0/3476602997 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbe0071090 con 0x7fcbe00aeda0 2024-03-29T10:40:44.582 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.592+0000 7fcbceffd640 1 -- 172.21.15.57:0/3476602997 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fcbd0001890 con 0x7fcbe00aeda0 2024-03-29T10:40:44.582 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.592+0000 7fcbceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:44.582 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.592+0000 7fcbceffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:44.582 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.592+0000 7fcbceffd640 10 monclient(hunting): dump: 2024-03-29T10:40:44.582 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:44.582 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:44.583 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:44.583 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:44.583 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:44.583 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:44.583 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:44.583 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:44.583 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:44.583 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:44.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.592+0000 7fcbceffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:44.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.592+0000 7fcbceffd640 10 client.?.objecter ms_handle_connect 0x7fcbe013c470 2024-03-29T10:40:44.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.592+0000 7fcbceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:44.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.592+0000 7fcbceffd640 1 -- 172.21.15.57:0/3476602997 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4083800962 0 0) 0x7fcbe0071090 con 0x7fcbe00aeda0 2024-03-29T10:40:44.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.592+0000 7fcbceffd640 1 -- 172.21.15.57:0/3476602997 >> v1:172.21.15.57:6790/0 conn(0x7fcbe013c470 legacy=0x7fcbe0138c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:44.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.593+0000 7fcbceffd640 1 -- 172.21.15.57:0/3476602997 >> v1:172.21.15.179:6789/0 conn(0x7fcbe006ee40 legacy=0x7fcbe01de950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:44.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.593+0000 7fcbceffd640 10 monclient: _finish_hunting 0 2024-03-29T10:40:44.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.593+0000 7fcbceffd640 1 monclient: found mon.a 2024-03-29T10:40:44.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.593+0000 7fcbceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:44.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.593+0000 7fcbceffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:40:44.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.593+0000 7fcbceffd640 1 -- 172.21.15.57:0/3476602997 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe0139330 con 0x7fcbe00aeda0 2024-03-29T10:40:44.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.593+0000 7fcbceffd640 10 monclient: _finish_auth 0 2024-03-29T10:40:44.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.593+0000 7fcbceffd640 10 monclient: _check_auth_tickets 2024-03-29T10:40:44.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.593+0000 7fcbceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:44.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.593+0000 7fcbe4aa8640 5 monclient: authenticate success, global_id 4260 2024-03-29T10:40:44.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.593+0000 7fcbe4aa8640 10 monclient: _renew_subs 2024-03-29T10:40:44.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.593+0000 7fcbe4aa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:40:44.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.593+0000 7fcbe4aa8640 1 -- 172.21.15.57:0/3476602997 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe00a9d60 con 0x7fcbe00aeda0 2024-03-29T10:40:44.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.593+0000 7fcbe4aa8640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:44.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.593+0000 7fcbe4aa8640 10 monclient: _renew_subs 2024-03-29T10:40:44.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.593+0000 7fcbe4aa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:40:44.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.593+0000 7fcbe4aa8640 1 -- 172.21.15.57:0/3476602997 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe00aa320 con 0x7fcbe00aeda0 2024-03-29T10:40:44.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.593+0000 7fcbe4aa8640 1 librados: init done 2024-03-29T10:40:44.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.594+0000 7fcbceffd640 1 -- 172.21.15.57:0/3476602997 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbd0001a60 con 0x7fcbe00aeda0 2024-03-29T10:40:44.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.594+0000 7fcbceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:44.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.594+0000 7fcbceffd640 1 -- 172.21.15.57:0/3476602997 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fcbd0003870 con 0x7fcbe00aeda0 2024-03-29T10:40:44.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.594+0000 7fcbceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:44.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.594+0000 7fcbceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:44.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.594+0000 7fcbceffd640 10 monclient: dump: 2024-03-29T10:40:44.585 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:44.585 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:44.585 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:44.585 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:44.585 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:44.586 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:44.586 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:44.586 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:44.586 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:44.586 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:44.587 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.595+0000 7fcbdcd99640 4 set_mon_vals no callback set 2024-03-29T10:40:44.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.596+0000 7fcbe4aa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:44.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.596+0000 7fcbceffd640 1 -- 172.21.15.57:0/3476602997 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fcbd00039d0 con 0x7fcbe00aeda0 2024-03-29T10:40:44.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.596+0000 7fcbe4aa8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:44.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.596+0000 7fcbe4aa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:40:44.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.596+0000 7fcbe4aa8640 1 -- 172.21.15.57:0/3476602997 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc4005080 con 0x7fcbe00aeda0 2024-03-29T10:40:44.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.597+0000 7fcbde59c640 10 monclient: get_auth_request con 0x7fcbc006dbb0 auth_method 0 2024-03-29T10:40:44.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.597+0000 7fcbceffd640 10 client.4260.objecter ms_handle_connect 0x7fcbc006dbb0 2024-03-29T10:40:44.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.597+0000 7fcbceffd640 1 -- 172.21.15.57:0/3476602997 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7fcbd00547d0 con 0x7fcbe00aeda0 2024-03-29T10:40:44.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.597+0000 7fcbceffd640 10 client.4260.objecter ms_dispatch 0x7fcbe0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:44.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.597+0000 7fcbceffd640 3 client.4260.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:44.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.597+0000 7fcbceffd640 3 client.4260.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:44.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.597+0000 7fcbceffd640 20 client.4260.objecter dump_active .. 0 homeless 2024-03-29T10:40:44.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.607+0000 7fcbceffd640 1 -- 172.21.15.57:0/3476602997 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcbd0050a70 con 0x7fcbe00aeda0 2024-03-29T10:40:44.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.607+0000 7fcbceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:44.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.607+0000 7fcbceffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:44.947 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.963+0000 7fcbe4aa8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:44.947 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.963+0000 7fcbe4aa8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:44.947 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.963+0000 7fcbe4aa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:40:44.947 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.963+0000 7fcbe4aa8640 1 -- 172.21.15.57:0/3476602997 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbc4005330 con 0x7fcbe00aeda0 2024-03-29T10:40:44.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.965+0000 7fcbceffd640 1 -- 172.21.15.57:0/3476602997 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7fcbd0050a70 con 0x7fcbe00aeda0 2024-03-29T10:40:44.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.965+0000 7fcbceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:44.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.965+0000 7fcbceffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:44.950 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:40:44.950 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:44.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.969+0000 7fcbe4aa8640 10 librados: watch_flush enter 2024-03-29T10:40:44.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.969+0000 7fcbe4aa8640 10 librados: watch_flush exit 2024-03-29T10:40:44.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.969+0000 7fcbe4aa8640 20 client.4260.objecter shutdown clearing up homeless session... 2024-03-29T10:40:44.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.969+0000 7fcbe4aa8640 10 client.4260.objecter successfully canceled tick 2024-03-29T10:40:44.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.969+0000 7fcbe4aa8640 1 -- 172.21.15.57:0/3476602997 >> v1:172.21.15.179:6816/2698295936 conn(0x7fcbc006dbb0 legacy=0x7fcbc0070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:44.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.969+0000 7fcbe4aa8640 10 monclient: shutdown 2024-03-29T10:40:44.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.969+0000 7fcbe4aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:44.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.969+0000 7fcbe4aa8640 1 -- 172.21.15.57:0/3476602997 >> v1:172.21.15.57:6789/0 conn(0x7fcbe00aeda0 legacy=0x7fcbe0138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:44.954 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.970+0000 7fcbe4aa8640 1 -- 172.21.15.57:0/3476602997 shutdown_connections 2024-03-29T10:40:44.954 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.970+0000 7fcbe4aa8640 1 -- 172.21.15.57:0/3476602997 >> 172.21.15.57:0/3476602997 conn(0x7fcbe0067470 msgr2=0x7fcbe0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:44.954 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.970+0000 7fcbe4aa8640 1 -- 172.21.15.57:0/3476602997 shutdown_connections 2024-03-29T10:40:44.954 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.970+0000 7fcbe4aa8640 1 -- 172.21.15.57:0/3476602997 wait complete. 2024-03-29T10:40:44.954 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:44.970+0000 7fcbe4aa8640 1 librados: shutdown 2024-03-29T10:40:44.975 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:51.781 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:51.933 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.950+0000 7f76eb356640 10 monclient: get_monmap_and_config 2024-03-29T10:40:51.934 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.950+0000 7f76eb356640 10 monclient: build_initial_monmap 2024-03-29T10:40:51.934 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.950+0000 7f76eb356640 10 monclient: monmap: 2024-03-29T10:40:51.934 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:40:51.934 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:51.934 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:40:51.950807+0000 2024-03-29T10:40:51.934 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:40:51.950807+0000 2024-03-29T10:40:51.934 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:51.934 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:51.934 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:40:51.934 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:40:51.934 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:40:51.934 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:51.935 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.951+0000 7f76eb356640 1 Processor -- start 2024-03-29T10:40:51.935 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.951+0000 7f76eb356640 1 -- start start 2024-03-29T10:40:51.935 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.951+0000 7f76eb356640 10 monclient: init 2024-03-29T10:40:51.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.952+0000 7f76eb356640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:51.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.952+0000 7f76eb356640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:51.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.952+0000 7f76eb356640 10 monclient(hunting): picked mon.noname-c con 0x7f76e40b4710 addr v1:172.21.15.179:6789/0 2024-03-29T10:40:51.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.952+0000 7f76eb356640 10 monclient(hunting): picked mon.noname-b con 0x7f76e40b3970 addr v1:172.21.15.57:6790/0 2024-03-29T10:40:51.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.952+0000 7f76eb356640 10 monclient(hunting): picked mon.noname-a con 0x7f76e413f390 addr v1:172.21.15.57:6789/0 2024-03-29T10:40:51.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.952+0000 7f76eb356640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e40af020 con 0x7f76e413f390 2024-03-29T10:40:51.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.952+0000 7f76eb356640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e40b0910 con 0x7f76e40b4710 2024-03-29T10:40:51.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.952+0000 7f76eb356640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e40b0bf0 con 0x7f76e40b3970 2024-03-29T10:40:51.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.952+0000 7f76eb356640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:51.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.952+0000 7f76eb356640 10 monclient(hunting): authenticate will time out at 1004.128750s 2024-03-29T10:40:51.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.952+0000 7f76e90f2640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f76e40b4710 0x7f76e40b1010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:60068/0 (socket says 172.21.15.57:60068) 2024-03-29T10:40:51.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.952+0000 7f76e90f2640 1 -- 172.21.15.57:0/910522464 learned_addr learned my addr 172.21.15.57:0/910522464 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:40:51.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.953+0000 7f76e3fff640 1 -- 172.21.15.57:0/910522464 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3823189525 0 0) 0x7f76e40b0910 con 0x7f76e40b4710 2024-03-29T10:40:51.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.953+0000 7f76e3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:51.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.953+0000 7f76e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:51.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.953+0000 7f76e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:51.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.953+0000 7f76e3fff640 10 monclient(hunting): my global_id is 4267 2024-03-29T10:40:51.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.953+0000 7f76e3fff640 1 -- 172.21.15.57:0/910522464 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76d4003520 con 0x7f76e40b4710 2024-03-29T10:40:51.941 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.953+0000 7f76e3fff640 1 -- 172.21.15.57:0/910522464 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2149828457 0 0) 0x7f76e40af020 con 0x7f76e413f390 2024-03-29T10:40:51.941 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.953+0000 7f76e3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:51.941 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.953+0000 7f76e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:51.941 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.953+0000 7f76e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:51.941 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.953+0000 7f76e3fff640 10 monclient(hunting): my global_id is 4263 2024-03-29T10:40:51.941 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.953+0000 7f76e3fff640 1 -- 172.21.15.57:0/910522464 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76e40b0910 con 0x7f76e413f390 2024-03-29T10:40:51.941 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.954+0000 7f76e3fff640 1 -- 172.21.15.57:0/910522464 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2396401556 0 0) 0x7f76d4003520 con 0x7f76e40b4710 2024-03-29T10:40:51.941 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.954+0000 7f76e3fff640 1 -- 172.21.15.57:0/910522464 >> v1:172.21.15.57:6790/0 conn(0x7f76e40b3970 legacy=0x7f76e40af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:40:51.941 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.954+0000 7f76e3fff640 1 -- 172.21.15.57:0/910522464 >> v1:172.21.15.57:6789/0 conn(0x7f76e413f390 legacy=0x7f76e413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:51.941 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.954+0000 7f76e3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:40:51.941 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.954+0000 7f76e3fff640 1 monclient: found mon.noname-c 2024-03-29T10:40:51.941 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.954+0000 7f76e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:51.941 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.954+0000 7f76e3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:40:51.941 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.954+0000 7f76e3fff640 1 -- 172.21.15.57:0/910522464 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e40aeb00 con 0x7f76e40b4710 2024-03-29T10:40:51.941 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.954+0000 7f76e3fff640 10 monclient: _finish_auth 0 2024-03-29T10:40:51.941 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.954+0000 7f76e3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:40:51.942 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.954+0000 7f76e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:51.942 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.954+0000 7f76eb356640 5 monclient: authenticate success, global_id 4267 2024-03-29T10:40:51.942 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.954+0000 7f76eb356640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:51.942 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.954+0000 7f76e3fff640 1 -- 172.21.15.57:0/910522464 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f76d8001a10 con 0x7f76e40b4710 2024-03-29T10:40:51.942 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.954+0000 7f76e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:51.942 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.954+0000 7f76e3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:51.947 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.954+0000 7f76e3fff640 10 monclient: dump: 2024-03-29T10:40:51.947 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:51.947 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:51.947 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:51.947 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:51.947 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:51.947 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:51.947 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:51.947 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:51.948 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:51.948 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:51.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.954+0000 7f76eb356640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:51.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.955+0000 7f76e3fff640 1 -- 172.21.15.57:0/910522464 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76d8001c30 con 0x7f76e40b4710 2024-03-29T10:40:51.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.955+0000 7f76e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:51.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.955+0000 7f76eb356640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:51.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.955+0000 7f76eb356640 4 set_mon_vals no callback set 2024-03-29T10:40:51.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.955+0000 7f76e3fff640 1 -- 172.21.15.57:0/910522464 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f76d8001fc0 con 0x7f76e40b4710 2024-03-29T10:40:51.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.955+0000 7f76e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:51.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.955+0000 7f76e3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:51.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.955+0000 7f76e3fff640 10 monclient: dump: 2024-03-29T10:40:51.948 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:51.948 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:51.948 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:51.949 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:51.949 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:51.949 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:51.949 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:51.949 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:51.949 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:51.949 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:51.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.956+0000 7f76eb356640 10 monclient: shutdown 2024-03-29T10:40:51.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.956+0000 7f76eb356640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:51.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.956+0000 7f76eb356640 1 -- 172.21.15.57:0/910522464 >> v1:172.21.15.179:6789/0 conn(0x7f76e40b4710 legacy=0x7f76e40b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:51.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.956+0000 7f76eb356640 1 -- 172.21.15.57:0/910522464 shutdown_connections 2024-03-29T10:40:51.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.956+0000 7f76eb356640 1 -- 172.21.15.57:0/910522464 >> 172.21.15.57:0/910522464 conn(0x7f76e4066a00 msgr2=0x7f76e4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:51.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.956+0000 7f76eb356640 1 -- 172.21.15.57:0/910522464 shutdown_connections 2024-03-29T10:40:51.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.956+0000 7f76eb356640 1 -- 172.21.15.57:0/910522464 wait complete. 2024-03-29T10:40:51.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.956+0000 7f76eb356640 10 monclient: build_initial_monmap 2024-03-29T10:40:51.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.956+0000 7f76eb356640 10 monclient: monmap: 2024-03-29T10:40:51.958 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:40:51.958 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:51.958 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:40:51.958 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:40:51.958 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:51.958 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:51.958 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:40:51.958 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:40:51.958 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:40:51.958 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:51.958 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.957+0000 7f76eb356640 1 librados: starting msgr at 2024-03-29T10:40:51.958 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.957+0000 7f76eb356640 1 librados: starting objecter 2024-03-29T10:40:51.958 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.957+0000 7f76eb356640 1 Processor -- start 2024-03-29T10:40:51.959 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.957+0000 7f76eb356640 1 -- start start 2024-03-29T10:40:51.959 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.957+0000 7f76eb356640 1 librados: setting wanted keys 2024-03-29T10:40:51.959 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.957+0000 7f76eb356640 1 librados: calling monclient init 2024-03-29T10:40:51.959 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.957+0000 7f76eb356640 10 monclient: init 2024-03-29T10:40:51.959 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.958+0000 7f76eb356640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:51.959 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.958+0000 7f76eb356640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:40:51.959 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.958+0000 7f76eb356640 10 monclient(hunting): picked mon.noname-c con 0x7f76e40b3970 addr v1:172.21.15.57:6790/0 2024-03-29T10:40:51.959 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.958+0000 7f76eb356640 10 monclient(hunting): picked mon.noname-a con 0x7f76e40b4710 addr v1:172.21.15.57:6789/0 2024-03-29T10:40:51.959 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.958+0000 7f76eb356640 10 monclient(hunting): picked mon.noname-b con 0x7f76e413f390 addr v1:172.21.15.179:6789/0 2024-03-29T10:40:51.959 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.958+0000 7f76eb356640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e4133650 con 0x7f76e40b4710 2024-03-29T10:40:51.959 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.958+0000 7f76eb356640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e406f4f0 con 0x7f76e413f390 2024-03-29T10:40:51.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.958+0000 7f76eb356640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e406f7d0 con 0x7f76e40b3970 2024-03-29T10:40:51.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.958+0000 7f76eb356640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:51.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.958+0000 7f76eb356640 10 monclient(hunting): authenticate will time out at 1004.135463s 2024-03-29T10:40:51.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.958+0000 7f76e88f1640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f76e40b4710 0x7f76e4132830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:48328/0 (socket says 172.21.15.57:48328) 2024-03-29T10:40:51.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.958+0000 7f76e98f3640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f76e413f390 0x7f76e4132f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:60826/0 (socket says 172.21.15.57:60826) 2024-03-29T10:40:51.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.958+0000 7f76e88f1640 1 -- 172.21.15.57:0/4195466558 learned_addr learned my addr 172.21.15.57:0/4195466558 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:40:51.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 10 client.?.objecter ms_handle_connect 0x7f76e413f390 2024-03-29T10:40:51.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:51.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 10 client.?.objecter ms_handle_connect 0x7f76e40b3970 2024-03-29T10:40:51.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:51.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 10 client.?.objecter ms_handle_connect 0x7f76e40b4710 2024-03-29T10:40:51.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:51.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 1 -- 172.21.15.57:0/4195466558 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2207803518 0 0) 0x7f76e406f4f0 con 0x7f76e413f390 2024-03-29T10:40:51.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:51.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:51.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:51.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 10 monclient(hunting): my global_id is 4270 2024-03-29T10:40:51.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 1 -- 172.21.15.57:0/4195466558 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76c8004eb0 con 0x7f76e413f390 2024-03-29T10:40:51.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 1 -- 172.21.15.57:0/4195466558 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2150083166 0 0) 0x7f76c8004eb0 con 0x7f76e413f390 2024-03-29T10:40:51.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 1 -- 172.21.15.57:0/4195466558 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76e406f4f0 con 0x7f76e413f390 2024-03-29T10:40:51.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 1 -- 172.21.15.57:0/4195466558 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f76dc0017e0 con 0x7f76e413f390 2024-03-29T10:40:51.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:51.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:51.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 10 monclient(hunting): dump: 2024-03-29T10:40:51.961 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:51.961 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:51.961 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:51.961 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:51.961 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:51.962 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:51.962 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:51.962 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:51.962 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:51.962 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:51.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:51.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 1 -- 172.21.15.57:0/4195466558 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285844471 0 0) 0x7f76e406f7d0 con 0x7f76e40b3970 2024-03-29T10:40:51.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:51.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:51.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:51.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 10 monclient(hunting): my global_id is 4283 2024-03-29T10:40:51.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.959+0000 7f76e1ffb640 1 -- 172.21.15.57:0/4195466558 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76c8004eb0 con 0x7f76e40b3970 2024-03-29T10:40:51.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.960+0000 7f76e1ffb640 1 -- 172.21.15.57:0/4195466558 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2096165719 0 0) 0x7f76e406f4f0 con 0x7f76e413f390 2024-03-29T10:40:51.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.960+0000 7f76e1ffb640 1 -- 172.21.15.57:0/4195466558 >> v1:172.21.15.57:6790/0 conn(0x7f76e40b3970 legacy=0x7f76e41db980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:51.963 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.960+0000 7f76e1ffb640 1 -- 172.21.15.57:0/4195466558 >> v1:172.21.15.57:6789/0 conn(0x7f76e40b4710 legacy=0x7f76e4132830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:51.963 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.960+0000 7f76e1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:40:51.963 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.960+0000 7f76e1ffb640 1 monclient: found mon.b 2024-03-29T10:40:51.963 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.960+0000 7f76e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:51.963 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.960+0000 7f76e1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:40:51.963 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.960+0000 7f76e1ffb640 1 -- 172.21.15.57:0/4195466558 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e406fab0 con 0x7f76e413f390 2024-03-29T10:40:51.963 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.960+0000 7f76e1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:40:51.963 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.960+0000 7f76e1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:40:51.963 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.960+0000 7f76e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:51.963 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.960+0000 7f76eb356640 5 monclient: authenticate success, global_id 4270 2024-03-29T10:40:51.963 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.960+0000 7f76eb356640 10 monclient: _renew_subs 2024-03-29T10:40:51.963 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.960+0000 7f76eb356640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:40:51.963 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.960+0000 7f76eb356640 1 -- 172.21.15.57:0/4195466558 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76e4135730 con 0x7f76e413f390 2024-03-29T10:40:51.963 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.960+0000 7f76eb356640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:51.963 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.960+0000 7f76eb356640 10 monclient: _renew_subs 2024-03-29T10:40:51.964 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.960+0000 7f76eb356640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:40:51.964 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.960+0000 7f76eb356640 1 -- 172.21.15.57:0/4195466558 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76e4138730 con 0x7f76e413f390 2024-03-29T10:40:51.964 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.960+0000 7f76eb356640 1 librados: init done 2024-03-29T10:40:51.964 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.961+0000 7f76e1ffb640 1 -- 172.21.15.57:0/4195466558 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76dc001940 con 0x7f76e413f390 2024-03-29T10:40:51.964 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.961+0000 7f76e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:51.964 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.961+0000 7f76e37fe640 4 set_mon_vals no callback set 2024-03-29T10:40:51.964 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.962+0000 7f76eb356640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:51.964 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.962+0000 7f76e1ffb640 1 -- 172.21.15.57:0/4195466558 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f76dc0037b0 con 0x7f76e413f390 2024-03-29T10:40:51.964 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.962+0000 7f76e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:51.964 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.962+0000 7f76e1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:51.964 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.962+0000 7f76e1ffb640 10 monclient: dump: 2024-03-29T10:40:51.964 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:51.964 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:51.964 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:51.964 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:51.965 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:51.965 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:51.965 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:51.965 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:51.965 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:51.965 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:51.965 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.962+0000 7f76eb356640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:51.965 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.962+0000 7f76eb356640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:40:51.965 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.962+0000 7f76eb356640 1 -- 172.21.15.57:0/4195466558 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76c4005080 con 0x7f76e413f390 2024-03-29T10:40:51.965 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.963+0000 7f76e1ffb640 1 -- 172.21.15.57:0/4195466558 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f76dc003910 con 0x7f76e413f390 2024-03-29T10:40:51.965 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.963+0000 7f76e1ffb640 1 -- 172.21.15.57:0/4195466558 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f76dc0848b0 con 0x7f76e413f390 2024-03-29T10:40:51.965 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.963+0000 7f76e1ffb640 10 client.4270.objecter ms_dispatch 0x7f76e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:51.965 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.963+0000 7f76e1ffb640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:51.965 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.963+0000 7f76e1ffb640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:51.965 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.964+0000 7f76e1ffb640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-29T10:40:51.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.964+0000 7f76e90f2640 10 monclient: get_auth_request con 0x7f76c806dd70 auth_method 0 2024-03-29T10:40:51.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.964+0000 7f76e1ffb640 10 client.4270.objecter ms_handle_connect 0x7f76c806dd70 2024-03-29T10:40:51.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.969+0000 7f76e1ffb640 1 -- 172.21.15.57:0/4195466558 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76dc050190 con 0x7f76e413f390 2024-03-29T10:40:51.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.969+0000 7f76e1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:51.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:51.969+0000 7f76e1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:52.382 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:52.398+0000 7f76eb356640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:52.382 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:52.398+0000 7f76eb356640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:52.383 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:52.398+0000 7f76eb356640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:40:52.383 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:52.398+0000 7f76eb356640 1 -- 172.21.15.57:0/4195466558 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76c4005330 con 0x7f76e413f390 2024-03-29T10:40:52.383 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:52.399+0000 7f76e1ffb640 1 -- 172.21.15.57:0/4195466558 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f76dc0520c0 con 0x7f76e413f390 2024-03-29T10:40:52.384 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:52.399+0000 7f76e1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:52.384 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:52.399+0000 7f76e1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:52.384 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:40:52.384 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:52.388 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:52.403+0000 7f76eb356640 10 librados: watch_flush enter 2024-03-29T10:40:52.388 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:52.403+0000 7f76eb356640 10 librados: watch_flush exit 2024-03-29T10:40:52.388 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:52.403+0000 7f76eb356640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-29T10:40:52.388 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:52.404+0000 7f76eb356640 10 client.4270.objecter successfully canceled tick 2024-03-29T10:40:52.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:52.404+0000 7f76eb356640 1 -- 172.21.15.57:0/4195466558 >> v1:172.21.15.179:6816/2698295936 conn(0x7f76c806dd70 legacy=0x7f76c8070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:52.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:52.404+0000 7f76eb356640 10 monclient: shutdown 2024-03-29T10:40:52.390 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:52.404+0000 7f76eb356640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:52.390 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:52.404+0000 7f76eb356640 1 -- 172.21.15.57:0/4195466558 >> v1:172.21.15.179:6789/0 conn(0x7f76e413f390 legacy=0x7f76e4132f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:52.390 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:52.404+0000 7f76eb356640 1 -- 172.21.15.57:0/4195466558 shutdown_connections 2024-03-29T10:40:52.390 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:52.404+0000 7f76eb356640 1 -- 172.21.15.57:0/4195466558 >> 172.21.15.57:0/4195466558 conn(0x7f76e4066a00 msgr2=0x7f76e4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:52.390 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:52.405+0000 7f76eb356640 1 -- 172.21.15.57:0/4195466558 shutdown_connections 2024-03-29T10:40:52.390 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:52.405+0000 7f76eb356640 1 -- 172.21.15.57:0/4195466558 wait complete. 2024-03-29T10:40:52.390 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:52.405+0000 7f76eb356640 1 librados: shutdown 2024-03-29T10:40:52.405 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:59.212 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:59.366 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.381+0000 7fc3092e4640 10 monclient: get_monmap_and_config 2024-03-29T10:40:59.367 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.381+0000 7fc3092e4640 10 monclient: build_initial_monmap 2024-03-29T10:40:59.367 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.381+0000 7fc3092e4640 10 monclient: monmap: 2024-03-29T10:40:59.367 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:40:59.367 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:59.367 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:40:59.383566+0000 2024-03-29T10:40:59.367 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:40:59.383566+0000 2024-03-29T10:40:59.367 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:59.367 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:59.367 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:40:59.367 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:40:59.367 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:40:59.367 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:59.368 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.382+0000 7fc3092e4640 1 Processor -- start 2024-03-29T10:40:59.368 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.382+0000 7fc3092e4640 1 -- start start 2024-03-29T10:40:59.368 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.382+0000 7fc3092e4640 10 monclient: init 2024-03-29T10:40:59.369 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.383+0000 7fc3092e4640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:59.369 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.383+0000 7fc3092e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:40:59.369 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.383+0000 7fc3092e4640 10 monclient(hunting): picked mon.noname-b con 0x7fc3040b0860 addr v1:172.21.15.57:6790/0 2024-03-29T10:40:59.369 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.383+0000 7fc3092e4640 10 monclient(hunting): picked mon.noname-c con 0x7fc30413ef30 addr v1:172.21.15.179:6789/0 2024-03-29T10:40:59.369 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.384+0000 7fc3092e4640 10 monclient(hunting): picked mon.noname-a con 0x7fc3040af8e0 addr v1:172.21.15.57:6789/0 2024-03-29T10:40:59.369 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.384+0000 7fc3092e4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3040b0c40 con 0x7fc3040af8e0 2024-03-29T10:40:59.369 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.384+0000 7fc3092e4640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3040aed10 con 0x7fc30413ef30 2024-03-29T10:40:59.369 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.384+0000 7fc3092e4640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3040afd50 con 0x7fc3040b0860 2024-03-29T10:40:59.369 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.384+0000 7fc3092e4640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:59.369 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.384+0000 7fc3092e4640 10 monclient(hunting): authenticate will time out at 1011.561249s 2024-03-29T10:40:59.369 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.384+0000 7fc3027fc640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fc30413ef30 0x7fc3040af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:59114/0 (socket says 172.21.15.57:59114) 2024-03-29T10:40:59.369 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.385+0000 7fc3027fc640 1 -- 172.21.15.57:0/1375333176 learned_addr learned my addr 172.21.15.57:0/1375333176 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:40:59.370 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.385+0000 7fc301ffb640 1 -- 172.21.15.57:0/1375333176 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629751226 0 0) 0x7fc3040aed10 con 0x7fc30413ef30 2024-03-29T10:40:59.370 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.385+0000 7fc301ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:59.370 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.385+0000 7fc301ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:59.370 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.385+0000 7fc301ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:59.370 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.385+0000 7fc301ffb640 10 monclient(hunting): my global_id is 4273 2024-03-29T10:40:59.370 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.385+0000 7fc301ffb640 1 -- 172.21.15.57:0/1375333176 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2ec003520 con 0x7fc30413ef30 2024-03-29T10:40:59.370 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.386+0000 7fc301ffb640 1 -- 172.21.15.57:0/1375333176 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1930050984 0 0) 0x7fc2ec003520 con 0x7fc30413ef30 2024-03-29T10:40:59.370 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.386+0000 7fc301ffb640 1 -- 172.21.15.57:0/1375333176 >> v1:172.21.15.57:6790/0 conn(0x7fc3040b0860 legacy=0x7fc30413e820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:59.373 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.387+0000 7fc301ffb640 1 -- 172.21.15.57:0/1375333176 >> v1:172.21.15.57:6789/0 conn(0x7fc3040af8e0 legacy=0x7fc304149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:40:59.373 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.387+0000 7fc301ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:40:59.373 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.387+0000 7fc301ffb640 1 monclient: found mon.noname-c 2024-03-29T10:40:59.373 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.387+0000 7fc301ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:59.373 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.387+0000 7fc301ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:40:59.373 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.387+0000 7fc301ffb640 1 -- 172.21.15.57:0/1375333176 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3040ab1b0 con 0x7fc30413ef30 2024-03-29T10:40:59.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.387+0000 7fc301ffb640 10 monclient: _finish_auth 0 2024-03-29T10:40:59.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.387+0000 7fc301ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:40:59.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.387+0000 7fc301ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:59.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.387+0000 7fc301ffb640 1 -- 172.21.15.57:0/1375333176 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fc2f8001690 con 0x7fc30413ef30 2024-03-29T10:40:59.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.388+0000 7fc301ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:59.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.388+0000 7fc301ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:59.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.388+0000 7fc301ffb640 10 monclient: dump: 2024-03-29T10:40:59.374 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:59.374 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:59.374 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:59.374 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:59.374 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:59.374 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:59.374 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:59.375 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:59.375 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:59.375 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:59.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.388+0000 7fc3092e4640 5 monclient: authenticate success, global_id 4273 2024-03-29T10:40:59.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.388+0000 7fc3092e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:59.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.388+0000 7fc301ffb640 1 -- 172.21.15.57:0/1375333176 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2f8001920 con 0x7fc30413ef30 2024-03-29T10:40:59.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.388+0000 7fc301ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:59.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.389+0000 7fc301ffb640 1 -- 172.21.15.57:0/1375333176 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fc2f8001cb0 con 0x7fc30413ef30 2024-03-29T10:40:59.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.389+0000 7fc301ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:59.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.389+0000 7fc301ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:59.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.389+0000 7fc301ffb640 10 monclient: dump: 2024-03-29T10:40:59.375 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:59.375 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:59.375 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:59.376 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:59.376 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:59.376 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:59.376 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:59.376 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:59.376 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:59.376 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:59.376 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.389+0000 7fc3092e4640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:59.376 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.389+0000 7fc3092e4640 4 set_mon_vals no callback set 2024-03-29T10:40:59.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.390+0000 7fc3092e4640 10 monclient: shutdown 2024-03-29T10:40:59.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.390+0000 7fc3092e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:59.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.390+0000 7fc3092e4640 1 -- 172.21.15.57:0/1375333176 >> v1:172.21.15.179:6789/0 conn(0x7fc30413ef30 legacy=0x7fc3040af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:59.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.390+0000 7fc3092e4640 1 -- 172.21.15.57:0/1375333176 shutdown_connections 2024-03-29T10:40:59.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.390+0000 7fc3092e4640 1 -- 172.21.15.57:0/1375333176 >> 172.21.15.57:0/1375333176 conn(0x7fc30406b660 msgr2=0x7fc30406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:59.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.390+0000 7fc3092e4640 1 -- 172.21.15.57:0/1375333176 shutdown_connections 2024-03-29T10:40:59.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.390+0000 7fc3092e4640 1 -- 172.21.15.57:0/1375333176 wait complete. 2024-03-29T10:40:59.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.391+0000 7fc3092e4640 10 monclient: build_initial_monmap 2024-03-29T10:40:59.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.391+0000 7fc3092e4640 10 monclient: monmap: 2024-03-29T10:40:59.378 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:40:59.379 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:59.379 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:40:59.379 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:40:59.379 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:59.379 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:59.379 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:40:59.379 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:40:59.379 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:40:59.379 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:59.379 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.391+0000 7fc3092e4640 1 librados: starting msgr at 2024-03-29T10:40:59.379 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.391+0000 7fc3092e4640 1 librados: starting objecter 2024-03-29T10:40:59.379 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.392+0000 7fc3092e4640 1 Processor -- start 2024-03-29T10:40:59.379 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.392+0000 7fc3092e4640 1 -- start start 2024-03-29T10:40:59.379 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.392+0000 7fc3092e4640 1 librados: setting wanted keys 2024-03-29T10:40:59.379 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.392+0000 7fc3092e4640 1 librados: calling monclient init 2024-03-29T10:40:59.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.392+0000 7fc3092e4640 10 monclient: init 2024-03-29T10:40:59.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.393+0000 7fc3092e4640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:59.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.393+0000 7fc3092e4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:40:59.385 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.395+0000 7fc3092e4640 10 monclient(hunting): picked mon.noname-a con 0x7fc3040af8e0 addr v1:172.21.15.57:6789/0 2024-03-29T10:40:59.385 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.395+0000 7fc3092e4640 10 monclient(hunting): picked mon.noname-b con 0x7fc3040b0860 addr v1:172.21.15.179:6789/0 2024-03-29T10:40:59.385 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.395+0000 7fc3092e4640 10 monclient(hunting): picked mon.noname-c con 0x7fc30413ef30 addr v1:172.21.15.57:6790/0 2024-03-29T10:40:59.386 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.395+0000 7fc3092e4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc30406ffa0 con 0x7fc3040af8e0 2024-03-29T10:40:59.386 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.395+0000 7fc3027fc640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fc3040b0860 0x7fc304070c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:59404/0 (socket says 172.21.15.57:59404) 2024-03-29T10:40:59.386 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.395+0000 7fc3027fc640 1 -- 172.21.15.57:0/2144313596 learned_addr learned my addr 172.21.15.57:0/2144313596 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:40:59.386 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.395+0000 7fc3092e4640 1 -- 172.21.15.57:0/2144313596 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc304070280 con 0x7fc3040b0860 2024-03-29T10:40:59.386 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.395+0000 7fc3092e4640 1 -- 172.21.15.57:0/2144313596 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc304070560 con 0x7fc30413ef30 2024-03-29T10:40:59.386 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.395+0000 7fc3092e4640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:59.386 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.395+0000 7fc3092e4640 10 monclient(hunting): authenticate will time out at 1011.572589s 2024-03-29T10:40:59.386 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.396+0000 7fc2eb7fe640 10 client.?.objecter ms_handle_connect 0x7fc3040b0860 2024-03-29T10:40:59.386 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.396+0000 7fc2eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:59.386 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.396+0000 7fc2eb7fe640 10 client.?.objecter ms_handle_connect 0x7fc3040af8e0 2024-03-29T10:40:59.386 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.396+0000 7fc2eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:59.386 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.396+0000 7fc2eb7fe640 1 -- 172.21.15.57:0/2144313596 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3016308840 0 0) 0x7fc304070280 con 0x7fc3040b0860 2024-03-29T10:40:59.386 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.396+0000 7fc2eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:59.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.396+0000 7fc2eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:59.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.396+0000 7fc2eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:59.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.396+0000 7fc2eb7fe640 10 monclient(hunting): my global_id is 4276 2024-03-29T10:40:59.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.396+0000 7fc2eb7fe640 1 -- 172.21.15.57:0/2144313596 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2f0004eb0 con 0x7fc3040b0860 2024-03-29T10:40:59.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.396+0000 7fc2eb7fe640 1 -- 172.21.15.57:0/2144313596 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1762941567 0 0) 0x7fc30406ffa0 con 0x7fc3040af8e0 2024-03-29T10:40:59.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.396+0000 7fc2eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:59.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.396+0000 7fc2eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:59.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.397+0000 7fc2eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:59.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.397+0000 7fc2eb7fe640 10 monclient(hunting): my global_id is 4269 2024-03-29T10:40:59.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.397+0000 7fc2eb7fe640 1 -- 172.21.15.57:0/2144313596 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc304070280 con 0x7fc3040af8e0 2024-03-29T10:40:59.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.397+0000 7fc2eb7fe640 1 -- 172.21.15.57:0/2144313596 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 760316450 0 0) 0x7fc2f0004eb0 con 0x7fc3040b0860 2024-03-29T10:40:59.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.397+0000 7fc2eb7fe640 1 -- 172.21.15.57:0/2144313596 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc30406ffa0 con 0x7fc3040b0860 2024-03-29T10:40:59.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.397+0000 7fc2eb7fe640 1 -- 172.21.15.57:0/2144313596 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fc2f8001650 con 0x7fc3040b0860 2024-03-29T10:40:59.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.397+0000 7fc2eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:59.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.397+0000 7fc2eb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:59.388 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.397+0000 7fc2eb7fe640 10 monclient(hunting): dump: 2024-03-29T10:40:59.388 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:59.388 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:59.388 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:59.388 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:59.388 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:59.388 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:59.388 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:59.388 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:59.388 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:59.388 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:59.388 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.398+0000 7fc2eb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:59.388 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.398+0000 7fc2eb7fe640 10 client.?.objecter ms_handle_connect 0x7fc30413ef30 2024-03-29T10:40:59.388 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.398+0000 7fc2eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:59.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.398+0000 7fc2eb7fe640 1 -- 172.21.15.57:0/2144313596 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2520012633 0 0) 0x7fc304070560 con 0x7fc30413ef30 2024-03-29T10:40:59.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.398+0000 7fc2eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:59.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.398+0000 7fc2eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:59.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.398+0000 7fc2eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:59.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.398+0000 7fc2eb7fe640 10 monclient(hunting): my global_id is 4286 2024-03-29T10:40:59.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.398+0000 7fc2eb7fe640 1 -- 172.21.15.57:0/2144313596 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2f0004eb0 con 0x7fc30413ef30 2024-03-29T10:40:59.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.399+0000 7fc2eb7fe640 1 -- 172.21.15.57:0/2144313596 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3990722021 0 0) 0x7fc304070280 con 0x7fc3040af8e0 2024-03-29T10:40:59.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.399+0000 7fc2eb7fe640 1 -- 172.21.15.57:0/2144313596 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc304070560 con 0x7fc3040af8e0 2024-03-29T10:40:59.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.399+0000 7fc2eb7fe640 1 -- 172.21.15.57:0/2144313596 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fc2f4001490 con 0x7fc3040af8e0 2024-03-29T10:40:59.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.399+0000 7fc2eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:59.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.399+0000 7fc2eb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:59.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.399+0000 7fc2eb7fe640 10 monclient(hunting): dump: 2024-03-29T10:40:59.389 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:59.389 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:59.389 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:59.390 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:59.390 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:59.390 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:59.390 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:59.390 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:59.390 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:59.390 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:59.390 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.399+0000 7fc2eb7fe640 1 -- 172.21.15.57:0/2144313596 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2531846515 0 0) 0x7fc2f0004eb0 con 0x7fc30413ef30 2024-03-29T10:40:59.390 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.399+0000 7fc2eb7fe640 1 -- 172.21.15.57:0/2144313596 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc304070280 con 0x7fc30413ef30 2024-03-29T10:40:59.390 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.399+0000 7fc2eb7fe640 1 -- 172.21.15.57:0/2144313596 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1423949810 0 0) 0x7fc304070560 con 0x7fc3040af8e0 2024-03-29T10:40:59.390 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.399+0000 7fc2eb7fe640 1 -- 172.21.15.57:0/2144313596 >> v1:172.21.15.57:6790/0 conn(0x7fc30413ef30 legacy=0x7fc304071340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:59.390 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.399+0000 7fc2eb7fe640 1 -- 172.21.15.57:0/2144313596 >> v1:172.21.15.179:6789/0 conn(0x7fc3040b0860 legacy=0x7fc304070c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:59.390 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.400+0000 7fc2eb7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:40:59.390 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.400+0000 7fc2eb7fe640 1 monclient: found mon.a 2024-03-29T10:40:59.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.400+0000 7fc2eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:59.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.400+0000 7fc2eb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:40:59.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.400+0000 7fc2eb7fe640 1 -- 172.21.15.57:0/2144313596 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc30406f290 con 0x7fc3040af8e0 2024-03-29T10:40:59.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.400+0000 7fc2eb7fe640 10 monclient: _finish_auth 0 2024-03-29T10:40:59.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.400+0000 7fc2eb7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:40:59.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.401+0000 7fc2eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:59.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.401+0000 7fc2eb7fe640 1 -- 172.21.15.57:0/2144313596 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2f40015f0 con 0x7fc3040af8e0 2024-03-29T10:40:59.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.401+0000 7fc2eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:59.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.401+0000 7fc2eb7fe640 1 -- 172.21.15.57:0/2144313596 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fc2f4001c40 con 0x7fc3040af8e0 2024-03-29T10:40:59.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.401+0000 7fc2eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:59.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.401+0000 7fc2eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:59.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.401+0000 7fc2eb7fe640 10 monclient: dump: 2024-03-29T10:40:59.391 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:40:59.391 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:40:59.391 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:59.392 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:40:59.392 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:59.392 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:40:59.392 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:40:59.392 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:40:59.392 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:40:59.392 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:40:59.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.401+0000 7fc3092e4640 5 monclient: authenticate success, global_id 4269 2024-03-29T10:40:59.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.401+0000 7fc3092e4640 10 monclient: _renew_subs 2024-03-29T10:40:59.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.401+0000 7fc3092e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:40:59.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.401+0000 7fc3092e4640 1 -- 172.21.15.57:0/2144313596 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc30406ef60 con 0x7fc3040af8e0 2024-03-29T10:40:59.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.401+0000 7fc3092e4640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:59.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.401+0000 7fc3092e4640 10 monclient: _renew_subs 2024-03-29T10:40:59.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.401+0000 7fc3092e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:40:59.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.401+0000 7fc3092e4640 1 -- 172.21.15.57:0/2144313596 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc30414fd40 con 0x7fc3040af8e0 2024-03-29T10:40:59.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.401+0000 7fc3092e4640 1 librados: init done 2024-03-29T10:40:59.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.402+0000 7fc3017fa640 4 set_mon_vals no callback set 2024-03-29T10:40:59.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.404+0000 7fc3092e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:59.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.404+0000 7fc3092e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:59.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.404+0000 7fc3092e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:40:59.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.404+0000 7fc3092e4640 1 -- 172.21.15.57:0/2144313596 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e4005080 con 0x7fc3040af8e0 2024-03-29T10:40:59.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.410+0000 7fc2eb7fe640 1 -- 172.21.15.57:0/2144313596 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fc2f40196b0 con 0x7fc3040af8e0 2024-03-29T10:40:59.395 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.410+0000 7fc2eb7fe640 1 -- 172.21.15.57:0/2144313596 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7fc2f40835d0 con 0x7fc3040af8e0 2024-03-29T10:40:59.395 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.410+0000 7fc2eb7fe640 10 client.4269.objecter ms_dispatch 0x7fc304000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:59.395 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.411+0000 7fc2eb7fe640 3 client.4269.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:59.395 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.411+0000 7fc2eb7fe640 3 client.4269.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:59.395 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.411+0000 7fc3027fc640 10 monclient: get_auth_request con 0x7fc2f006e100 auth_method 0 2024-03-29T10:40:59.395 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.411+0000 7fc2eb7fe640 20 client.4269.objecter dump_active .. 0 homeless 2024-03-29T10:40:59.395 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.411+0000 7fc2eb7fe640 10 client.4269.objecter ms_handle_connect 0x7fc2f006e100 2024-03-29T10:40:59.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.413+0000 7fc2eb7fe640 1 -- 172.21.15.57:0/2144313596 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc2f404eeb0 con 0x7fc3040af8e0 2024-03-29T10:40:59.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.413+0000 7fc2eb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:59.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.413+0000 7fc2eb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:59.740 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.755+0000 7fc3092e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:59.741 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.755+0000 7fc3092e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:59.741 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.755+0000 7fc3092e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:40:59.741 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.755+0000 7fc3092e4640 1 -- 172.21.15.57:0/2144313596 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2e4005330 con 0x7fc3040af8e0 2024-03-29T10:40:59.741 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.757+0000 7fc2eb7fe640 1 -- 172.21.15.57:0/2144313596 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7fc2f4050de0 con 0x7fc3040af8e0 2024-03-29T10:40:59.741 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.757+0000 7fc2eb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:59.742 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.757+0000 7fc2eb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:59.742 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:40:59.742 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:59.746 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.761+0000 7fc3092e4640 10 librados: watch_flush enter 2024-03-29T10:40:59.746 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.761+0000 7fc3092e4640 10 librados: watch_flush exit 2024-03-29T10:40:59.746 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.761+0000 7fc3092e4640 20 client.4269.objecter shutdown clearing up homeless session... 2024-03-29T10:40:59.746 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.761+0000 7fc3092e4640 10 client.4269.objecter successfully canceled tick 2024-03-29T10:40:59.746 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.761+0000 7fc3092e4640 1 -- 172.21.15.57:0/2144313596 >> v1:172.21.15.179:6816/2698295936 conn(0x7fc2f006e100 legacy=0x7fc2f00705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:59.746 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.761+0000 7fc3092e4640 10 monclient: shutdown 2024-03-29T10:40:59.746 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.761+0000 7fc3092e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:59.746 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.761+0000 7fc3092e4640 1 -- 172.21.15.57:0/2144313596 >> v1:172.21.15.57:6789/0 conn(0x7fc3040af8e0 legacy=0x7fc30406e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:59.746 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.761+0000 7fc3092e4640 1 -- 172.21.15.57:0/2144313596 shutdown_connections 2024-03-29T10:40:59.746 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.762+0000 7fc3092e4640 1 -- 172.21.15.57:0/2144313596 >> 172.21.15.57:0/2144313596 conn(0x7fc30406b660 msgr2=0x7fc30406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:59.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.762+0000 7fc3092e4640 1 -- 172.21.15.57:0/2144313596 shutdown_connections 2024-03-29T10:40:59.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.762+0000 7fc3092e4640 1 -- 172.21.15.57:0/2144313596 wait complete. 2024-03-29T10:40:59.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:40:59.762+0000 7fc3092e4640 1 librados: shutdown 2024-03-29T10:40:59.762 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:06.570 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:06.720 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.735+0000 7f8eb0b52640 10 monclient: get_monmap_and_config 2024-03-29T10:41:06.721 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.735+0000 7f8eb0b52640 10 monclient: build_initial_monmap 2024-03-29T10:41:06.721 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.735+0000 7f8eb0b52640 10 monclient: monmap: 2024-03-29T10:41:06.721 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:41:06.721 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:06.721 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:41:06.737521+0000 2024-03-29T10:41:06.721 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:41:06.737521+0000 2024-03-29T10:41:06.721 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:06.721 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:06.721 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:41:06.722 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:41:06.722 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:41:06.722 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:06.723 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.737+0000 7f8eb0b52640 1 Processor -- start 2024-03-29T10:41:06.723 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.737+0000 7f8eb0b52640 1 -- start start 2024-03-29T10:41:06.723 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.737+0000 7f8eb0b52640 10 monclient: init 2024-03-29T10:41:06.723 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.738+0000 7f8eb0b52640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:06.723 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.738+0000 7f8eb0b52640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:41:06.723 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.738+0000 7f8eb0b52640 10 monclient(hunting): picked mon.noname-b con 0x7f8eac0b0860 addr v1:172.21.15.57:6790/0 2024-03-29T10:41:06.723 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.738+0000 7f8eb0b52640 10 monclient(hunting): picked mon.noname-c con 0x7f8eac13ef30 addr v1:172.21.15.179:6789/0 2024-03-29T10:41:06.724 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.739+0000 7f8ea3fff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f8eac13ef30 0x7f8eac0af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:33020/0 (socket says 172.21.15.57:33020) 2024-03-29T10:41:06.724 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.740+0000 7f8eb0b52640 10 monclient(hunting): picked mon.noname-a con 0x7f8eac0af8e0 addr v1:172.21.15.57:6789/0 2024-03-29T10:41:06.724 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.740+0000 7f8eb0b52640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eac0b0c40 con 0x7f8eac0af8e0 2024-03-29T10:41:06.724 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.740+0000 7f8eb0b52640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eac0aed10 con 0x7f8eac13ef30 2024-03-29T10:41:06.725 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.740+0000 7f8eb0b52640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eac0afd50 con 0x7f8eac0b0860 2024-03-29T10:41:06.725 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.740+0000 7f8eb0b52640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:06.725 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.740+0000 7f8eb0b52640 10 monclient(hunting): authenticate will time out at 1018.917148s 2024-03-29T10:41:06.725 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.740+0000 7f8ea3fff640 1 -- 172.21.15.57:0/422369133 learned_addr learned my addr 172.21.15.57:0/422369133 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:41:06.728 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.741+0000 7f8ea9d9b640 1 -- 172.21.15.57:0/422369133 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806465427 0 0) 0x7f8eac0aed10 con 0x7f8eac13ef30 2024-03-29T10:41:06.728 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.741+0000 7f8ea9d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:41:06.728 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.741+0000 7f8ea9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:06.728 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.741+0000 7f8ea9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:06.728 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.741+0000 7f8ea9d9b640 10 monclient(hunting): my global_id is 4279 2024-03-29T10:41:06.729 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.741+0000 7f8ea9d9b640 1 -- 172.21.15.57:0/422369133 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e98003520 con 0x7f8eac13ef30 2024-03-29T10:41:06.729 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.741+0000 7f8ea9d9b640 1 -- 172.21.15.57:0/422369133 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404882678 0 0) 0x7f8eac0b0c40 con 0x7f8eac0af8e0 2024-03-29T10:41:06.729 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.741+0000 7f8ea9d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:41:06.729 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.741+0000 7f8ea9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:06.729 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.741+0000 7f8ea9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:06.729 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.741+0000 7f8ea9d9b640 10 monclient(hunting): my global_id is 4272 2024-03-29T10:41:06.729 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.741+0000 7f8ea9d9b640 1 -- 172.21.15.57:0/422369133 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eac0aed10 con 0x7f8eac0af8e0 2024-03-29T10:41:06.729 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8ea9d9b640 1 -- 172.21.15.57:0/422369133 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 58071606 0 0) 0x7f8e98003520 con 0x7f8eac13ef30 2024-03-29T10:41:06.729 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8ea9d9b640 1 -- 172.21.15.57:0/422369133 >> v1:172.21.15.57:6790/0 conn(0x7f8eac0b0860 legacy=0x7f8eac13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:41:06.729 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8ea9d9b640 1 -- 172.21.15.57:0/422369133 >> v1:172.21.15.57:6789/0 conn(0x7f8eac0af8e0 legacy=0x7f8eac149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:06.729 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8ea9d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:41:06.730 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8ea9d9b640 1 monclient: found mon.noname-c 2024-03-29T10:41:06.730 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8ea9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:06.730 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8ea9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:41:06.730 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8ea9d9b640 1 -- 172.21.15.57:0/422369133 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eac0ab1b0 con 0x7f8eac13ef30 2024-03-29T10:41:06.730 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8ea9d9b640 10 monclient: _finish_auth 0 2024-03-29T10:41:06.730 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8ea9d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:41:06.730 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8ea9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:06.730 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8ea9d9b640 1 -- 172.21.15.57:0/422369133 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f8e94001690 con 0x7f8eac13ef30 2024-03-29T10:41:06.730 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8ea9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:06.730 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8ea9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:06.730 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8ea9d9b640 10 monclient: dump: 2024-03-29T10:41:06.730 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:06.730 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:06.730 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:06.730 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:06.731 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:06.731 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:06.731 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:06.731 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:06.731 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:06.731 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:06.731 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8eb0b52640 5 monclient: authenticate success, global_id 4279 2024-03-29T10:41:06.731 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8eb0b52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:06.731 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8ea9d9b640 1 -- 172.21.15.57:0/422369133 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e940017f0 con 0x7f8eac13ef30 2024-03-29T10:41:06.731 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8ea9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:06.731 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8ea9d9b640 1 -- 172.21.15.57:0/422369133 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f8e94001c00 con 0x7f8eac13ef30 2024-03-29T10:41:06.731 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8ea9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:06.731 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8ea9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:06.731 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8ea9d9b640 10 monclient: dump: 2024-03-29T10:41:06.731 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:06.732 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:06.732 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:06.732 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:06.732 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:06.732 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:06.732 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:06.732 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:06.732 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:06.732 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:06.732 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8eb0b52640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:06.732 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.742+0000 7f8eb0b52640 4 set_mon_vals no callback set 2024-03-29T10:41:06.732 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.743+0000 7f8eb0b52640 10 monclient: shutdown 2024-03-29T10:41:06.732 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.743+0000 7f8eb0b52640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:06.732 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.743+0000 7f8eb0b52640 1 -- 172.21.15.57:0/422369133 >> v1:172.21.15.179:6789/0 conn(0x7f8eac13ef30 legacy=0x7f8eac0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:06.732 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.743+0000 7f8eb0b52640 1 -- 172.21.15.57:0/422369133 shutdown_connections 2024-03-29T10:41:06.732 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.743+0000 7f8eb0b52640 1 -- 172.21.15.57:0/422369133 >> 172.21.15.57:0/422369133 conn(0x7f8eac06b660 msgr2=0x7f8eac06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:06.733 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.743+0000 7f8eb0b52640 1 -- 172.21.15.57:0/422369133 shutdown_connections 2024-03-29T10:41:06.733 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.743+0000 7f8eb0b52640 1 -- 172.21.15.57:0/422369133 wait complete. 2024-03-29T10:41:06.742 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.745+0000 7f8eb0b52640 10 monclient: build_initial_monmap 2024-03-29T10:41:06.742 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.745+0000 7f8eb0b52640 10 monclient: monmap: 2024-03-29T10:41:06.742 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:41:06.742 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:06.742 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:41:06.742 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:41:06.742 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:06.742 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:06.742 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:41:06.742 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:41:06.742 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:41:06.742 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:06.742 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.745+0000 7f8eb0b52640 1 librados: starting msgr at 2024-03-29T10:41:06.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.745+0000 7f8eb0b52640 1 librados: starting objecter 2024-03-29T10:41:06.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.746+0000 7f8eb0b52640 1 Processor -- start 2024-03-29T10:41:06.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.746+0000 7f8eb0b52640 1 -- start start 2024-03-29T10:41:06.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.746+0000 7f8eb0b52640 1 librados: setting wanted keys 2024-03-29T10:41:06.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.746+0000 7f8eb0b52640 1 librados: calling monclient init 2024-03-29T10:41:06.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.746+0000 7f8eb0b52640 10 monclient: init 2024-03-29T10:41:06.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.747+0000 7f8eb0b52640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:06.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.747+0000 7f8eb0b52640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:41:06.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.747+0000 7f8eb0b52640 10 monclient(hunting): picked mon.noname-a con 0x7f8eac0af8e0 addr v1:172.21.15.57:6789/0 2024-03-29T10:41:06.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.747+0000 7f8eb0b52640 10 monclient(hunting): picked mon.noname-b con 0x7f8eac0b0860 addr v1:172.21.15.179:6789/0 2024-03-29T10:41:06.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.748+0000 7f8eb0b52640 10 monclient(hunting): picked mon.noname-c con 0x7f8eac13ef30 addr v1:172.21.15.57:6790/0 2024-03-29T10:41:06.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.748+0000 7f8eb0b52640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eac0b3990 con 0x7f8eac0af8e0 2024-03-29T10:41:06.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.748+0000 7f8eb0b52640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eac0b3c70 con 0x7f8eac0b0860 2024-03-29T10:41:06.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.748+0000 7f8eb0b52640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eac0b2660 con 0x7f8eac13ef30 2024-03-29T10:41:06.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.748+0000 7f8eb0b52640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:06.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.748+0000 7f8eb0b52640 10 monclient(hunting): authenticate will time out at 1018.925488s 2024-03-29T10:41:06.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.748+0000 7f8ea3fff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f8eac0b0860 0x7f8eac132330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:33246/0 (socket says 172.21.15.57:33246) 2024-03-29T10:41:06.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.748+0000 7f8ea3fff640 1 -- 172.21.15.57:0/2298014778 learned_addr learned my addr 172.21.15.57:0/2298014778 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:41:06.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.749+0000 7f8ea2ffd640 10 client.?.objecter ms_handle_connect 0x7f8eac0b0860 2024-03-29T10:41:06.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.749+0000 7f8ea2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:06.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.749+0000 7f8ea2ffd640 1 -- 172.21.15.57:0/2298014778 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266375333 0 0) 0x7f8eac0b3c70 con 0x7f8eac0b0860 2024-03-29T10:41:06.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.749+0000 7f8ea2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:06.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.749+0000 7f8ea2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:06.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.749+0000 7f8ea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:06.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.749+0000 7f8ea2ffd640 10 monclient(hunting): my global_id is 4282 2024-03-29T10:41:06.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.749+0000 7f8ea2ffd640 1 -- 172.21.15.57:0/2298014778 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e8c004eb0 con 0x7f8eac0b0860 2024-03-29T10:41:06.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.750+0000 7f8ea2ffd640 1 -- 172.21.15.57:0/2298014778 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1703502689 0 0) 0x7f8e8c004eb0 con 0x7f8eac0b0860 2024-03-29T10:41:06.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.750+0000 7f8ea2ffd640 1 -- 172.21.15.57:0/2298014778 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8eac0b3c70 con 0x7f8eac0b0860 2024-03-29T10:41:06.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.750+0000 7f8ea2ffd640 1 -- 172.21.15.57:0/2298014778 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f8e94001690 con 0x7f8eac0b0860 2024-03-29T10:41:06.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.751+0000 7f8ea2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:06.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.751+0000 7f8ea2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:06.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.751+0000 7f8ea2ffd640 10 monclient(hunting): dump: 2024-03-29T10:41:06.745 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:06.745 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:06.745 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:06.745 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:06.746 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:06.746 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:06.746 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:06.746 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:06.746 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:06.746 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:06.746 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.751+0000 7f8ea2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:06.746 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.751+0000 7f8ea2ffd640 10 client.?.objecter ms_handle_connect 0x7f8eac13ef30 2024-03-29T10:41:06.746 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.751+0000 7f8ea2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:06.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.751+0000 7f8ea2ffd640 10 client.?.objecter ms_handle_connect 0x7f8eac0af8e0 2024-03-29T10:41:06.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.751+0000 7f8ea2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:06.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.751+0000 7f8ea2ffd640 1 -- 172.21.15.57:0/2298014778 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204494994 0 0) 0x7f8eac0b2660 con 0x7f8eac13ef30 2024-03-29T10:41:06.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.751+0000 7f8ea2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:06.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.751+0000 7f8ea2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:06.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.751+0000 7f8ea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:06.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.751+0000 7f8ea2ffd640 10 monclient(hunting): my global_id is 4289 2024-03-29T10:41:06.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.751+0000 7f8ea2ffd640 1 -- 172.21.15.57:0/2298014778 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e8c004eb0 con 0x7f8eac13ef30 2024-03-29T10:41:06.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.751+0000 7f8ea2ffd640 1 -- 172.21.15.57:0/2298014778 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363073731 0 0) 0x7f8eac0b3990 con 0x7f8eac0af8e0 2024-03-29T10:41:06.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.751+0000 7f8ea2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:06.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.751+0000 7f8ea2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:06.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.751+0000 7f8ea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:06.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.752+0000 7f8ea2ffd640 10 monclient(hunting): my global_id is 4275 2024-03-29T10:41:06.748 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.752+0000 7f8ea2ffd640 1 -- 172.21.15.57:0/2298014778 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eac0b2660 con 0x7f8eac0af8e0 2024-03-29T10:41:06.748 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.752+0000 7f8ea2ffd640 1 -- 172.21.15.57:0/2298014778 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1488712277 0 0) 0x7f8eac0b3c70 con 0x7f8eac0b0860 2024-03-29T10:41:06.748 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.752+0000 7f8ea2ffd640 1 -- 172.21.15.57:0/2298014778 >> v1:172.21.15.57:6790/0 conn(0x7f8eac13ef30 legacy=0x7f8eac132a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:06.748 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.752+0000 7f8ea2ffd640 1 -- 172.21.15.57:0/2298014778 >> v1:172.21.15.57:6789/0 conn(0x7f8eac0af8e0 legacy=0x7f8eac1d2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:06.748 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.752+0000 7f8ea2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:41:06.748 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.752+0000 7f8ea2ffd640 1 monclient: found mon.b 2024-03-29T10:41:06.748 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.752+0000 7f8ea2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:06.748 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.752+0000 7f8ea2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:06.748 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.752+0000 7f8ea2ffd640 1 -- 172.21.15.57:0/2298014778 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eac0b2b80 con 0x7f8eac0b0860 2024-03-29T10:41:06.749 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.752+0000 7f8ea2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:41:06.749 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.752+0000 7f8ea2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:41:06.749 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.752+0000 7f8ea2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:06.749 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.752+0000 7f8eb0b52640 5 monclient: authenticate success, global_id 4282 2024-03-29T10:41:06.749 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.752+0000 7f8eb0b52640 10 monclient: _renew_subs 2024-03-29T10:41:06.749 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.753+0000 7f8eb0b52640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:06.749 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.753+0000 7f8eb0b52640 1 -- 172.21.15.57:0/2298014778 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eac071a10 con 0x7f8eac0b0860 2024-03-29T10:41:06.749 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.753+0000 7f8eb0b52640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:06.749 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.753+0000 7f8eb0b52640 10 monclient: _renew_subs 2024-03-29T10:41:06.749 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.753+0000 7f8eb0b52640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:06.749 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.753+0000 7f8eb0b52640 1 -- 172.21.15.57:0/2298014778 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eac0b4270 con 0x7f8eac0b0860 2024-03-29T10:41:06.749 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.753+0000 7f8eb0b52640 1 librados: init done 2024-03-29T10:41:06.749 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.753+0000 7f8ea2ffd640 1 -- 172.21.15.57:0/2298014778 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e94004a90 con 0x7f8eac0b0860 2024-03-29T10:41:06.750 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.753+0000 7f8ea2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:06.750 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.753+0000 7f8ea2ffd640 1 -- 172.21.15.57:0/2298014778 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f8e94004e60 con 0x7f8eac0b0860 2024-03-29T10:41:06.750 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.753+0000 7f8ea8d99640 4 set_mon_vals no callback set 2024-03-29T10:41:06.750 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.753+0000 7f8ea2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:06.750 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.753+0000 7f8ea2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:06.750 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.753+0000 7f8ea2ffd640 10 monclient: dump: 2024-03-29T10:41:06.750 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:06.750 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:06.750 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:06.750 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:06.751 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:06.751 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:06.751 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:06.751 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:06.751 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:06.751 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:06.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.755+0000 7f8eb0b52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:06.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.755+0000 7f8eb0b52640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:06.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.755+0000 7f8eb0b52640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:06.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.755+0000 7f8ea2ffd640 1 -- 172.21.15.57:0/2298014778 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f8e94004fc0 con 0x7f8eac0b0860 2024-03-29T10:41:06.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.755+0000 7f8eb0b52640 1 -- 172.21.15.57:0/2298014778 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e90005080 con 0x7f8eac0b0860 2024-03-29T10:41:06.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.755+0000 7f8ea2ffd640 1 -- 172.21.15.57:0/2298014778 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f8e94084bf0 con 0x7f8eac0b0860 2024-03-29T10:41:06.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.755+0000 7f8ea2ffd640 10 client.4282.objecter ms_dispatch 0x7f8eac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:06.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.755+0000 7f8ea2ffd640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:06.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.755+0000 7f8ea2ffd640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:06.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.755+0000 7f8ea2ffd640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-29T10:41:06.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.755+0000 7f8eaa59c640 10 monclient: get_auth_request con 0x7f8e8c06dc30 auth_method 0 2024-03-29T10:41:06.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.756+0000 7f8ea2ffd640 10 client.4282.objecter ms_handle_connect 0x7f8e8c06dc30 2024-03-29T10:41:06.753 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.765+0000 7f8ea2ffd640 1 -- 172.21.15.57:0/2298014778 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e940504d0 con 0x7f8eac0b0860 2024-03-29T10:41:06.753 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.765+0000 7f8ea2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:06.753 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:06.765+0000 7f8ea2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:07.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:07.108+0000 7f8eb0b52640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:07.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:07.108+0000 7f8eb0b52640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:07.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:07.108+0000 7f8eb0b52640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:07.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:07.108+0000 7f8eb0b52640 1 -- 172.21.15.57:0/2298014778 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e90005330 con 0x7f8eac0b0860 2024-03-29T10:41:07.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:07.109+0000 7f8ea2ffd640 1 -- 172.21.15.57:0/2298014778 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f8e94052400 con 0x7f8eac0b0860 2024-03-29T10:41:07.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:07.109+0000 7f8ea2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:07.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:07.109+0000 7f8ea2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:07.094 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:41:07.094 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:07.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:07.113+0000 7f8eb0b52640 10 librados: watch_flush enter 2024-03-29T10:41:07.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:07.113+0000 7f8eb0b52640 10 librados: watch_flush exit 2024-03-29T10:41:07.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:07.114+0000 7f8eb0b52640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-29T10:41:07.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:07.114+0000 7f8eb0b52640 10 client.4282.objecter successfully canceled tick 2024-03-29T10:41:07.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:07.114+0000 7f8eb0b52640 1 -- 172.21.15.57:0/2298014778 >> v1:172.21.15.179:6816/2698295936 conn(0x7f8e8c06dc30 legacy=0x7f8e8c0700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:07.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:07.114+0000 7f8eb0b52640 10 monclient: shutdown 2024-03-29T10:41:07.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:07.114+0000 7f8eb0b52640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:07.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:07.114+0000 7f8eb0b52640 1 -- 172.21.15.57:0/2298014778 >> v1:172.21.15.179:6789/0 conn(0x7f8eac0b0860 legacy=0x7f8eac132330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:07.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:07.114+0000 7f8eb0b52640 1 -- 172.21.15.57:0/2298014778 shutdown_connections 2024-03-29T10:41:07.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:07.114+0000 7f8eb0b52640 1 -- 172.21.15.57:0/2298014778 >> 172.21.15.57:0/2298014778 conn(0x7f8eac06b660 msgr2=0x7f8eac13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:07.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:07.114+0000 7f8eb0b52640 1 -- 172.21.15.57:0/2298014778 shutdown_connections 2024-03-29T10:41:07.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:07.115+0000 7f8eb0b52640 1 -- 172.21.15.57:0/2298014778 wait complete. 2024-03-29T10:41:07.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:07.115+0000 7f8eb0b52640 1 librados: shutdown 2024-03-29T10:41:07.116 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:13.922 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:14.070 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.086+0000 7fc32cc1b640 10 monclient: get_monmap_and_config 2024-03-29T10:41:14.071 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.086+0000 7fc32cc1b640 10 monclient: build_initial_monmap 2024-03-29T10:41:14.071 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.086+0000 7fc32cc1b640 10 monclient: monmap: 2024-03-29T10:41:14.071 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:41:14.071 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:14.071 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:41:14.086874+0000 2024-03-29T10:41:14.071 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:41:14.086874+0000 2024-03-29T10:41:14.071 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:14.071 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:14.071 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:41:14.072 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:41:14.072 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:41:14.072 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:14.072 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.087+0000 7fc32cc1b640 1 Processor -- start 2024-03-29T10:41:14.072 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.087+0000 7fc32cc1b640 1 -- start start 2024-03-29T10:41:14.072 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.087+0000 7fc32cc1b640 10 monclient: init 2024-03-29T10:41:14.072 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.088+0000 7fc32cc1b640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:14.072 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.088+0000 7fc32cc1b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:41:14.073 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.088+0000 7fc32cc1b640 10 monclient(hunting): picked mon.noname-a con 0x7fc3280a9750 addr v1:172.21.15.57:6789/0 2024-03-29T10:41:14.073 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.088+0000 7fc32cc1b640 10 monclient(hunting): picked mon.noname-b con 0x7fc3280ae220 addr v1:172.21.15.57:6790/0 2024-03-29T10:41:14.073 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.088+0000 7fc32cc1b640 10 monclient(hunting): picked mon.noname-c con 0x7fc32813b9d0 addr v1:172.21.15.179:6789/0 2024-03-29T10:41:14.073 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.088+0000 7fc32cc1b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc328142490 con 0x7fc3280a9750 2024-03-29T10:41:14.073 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.088+0000 7fc32cc1b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3280ac130 con 0x7fc32813b9d0 2024-03-29T10:41:14.073 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.088+0000 7fc32cc1b640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3280ac410 con 0x7fc3280ae220 2024-03-29T10:41:14.073 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.088+0000 7fc32cc1b640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:14.073 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.088+0000 7fc32cc1b640 10 monclient(hunting): authenticate will time out at 1026.264230s 2024-03-29T10:41:14.073 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.090+0000 7fc326ffd640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7fc3280ae220 0x7fc32813b2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:56679/0 (socket says 172.21.15.57:56679) 2024-03-29T10:41:14.074 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.090+0000 7fc326ffd640 1 -- 172.21.15.57:0/3919392863 learned_addr learned my addr 172.21.15.57:0/3919392863 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:41:14.076 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.091+0000 7fc3267fc640 1 -- 172.21.15.57:0/3919392863 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930985988 0 0) 0x7fc3280ac410 con 0x7fc3280ae220 2024-03-29T10:41:14.076 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.091+0000 7fc3267fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:14.077 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.091+0000 7fc3267fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:14.077 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.091+0000 7fc3267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:14.077 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.091+0000 7fc3267fc640 10 monclient(hunting): my global_id is 4292 2024-03-29T10:41:14.077 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.091+0000 7fc3267fc640 1 -- 172.21.15.57:0/3919392863 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc314003520 con 0x7fc3280ae220 2024-03-29T10:41:14.077 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.091+0000 7fc3267fc640 1 -- 172.21.15.57:0/3919392863 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 390598217 0 0) 0x7fc314003520 con 0x7fc3280ae220 2024-03-29T10:41:14.077 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.091+0000 7fc3267fc640 1 -- 172.21.15.57:0/3919392863 >> v1:172.21.15.179:6789/0 conn(0x7fc32813b9d0 legacy=0x7fc328149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:14.077 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.091+0000 7fc3267fc640 1 -- 172.21.15.57:0/3919392863 >> v1:172.21.15.57:6789/0 conn(0x7fc3280a9750 legacy=0x7fc328141d80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:14.077 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.091+0000 7fc3267fc640 10 monclient: _finish_hunting 0 2024-03-29T10:41:14.077 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.091+0000 7fc3267fc640 1 monclient: found mon.noname-b 2024-03-29T10:41:14.077 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.091+0000 7fc3267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:14.077 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.091+0000 7fc3267fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:41:14.077 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.091+0000 7fc3267fc640 1 -- 172.21.15.57:0/3919392863 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3280afa40 con 0x7fc3280ae220 2024-03-29T10:41:14.077 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.092+0000 7fc3267fc640 10 monclient: _finish_auth 0 2024-03-29T10:41:14.078 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.092+0000 7fc3267fc640 10 monclient: _check_auth_tickets 2024-03-29T10:41:14.078 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.092+0000 7fc3267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:14.078 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.092+0000 7fc3267fc640 1 -- 172.21.15.57:0/3919392863 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fc318001a10 con 0x7fc3280ae220 2024-03-29T10:41:14.078 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.092+0000 7fc3267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:14.078 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.092+0000 7fc3267fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:14.078 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.092+0000 7fc3267fc640 10 monclient: dump: 2024-03-29T10:41:14.078 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:14.078 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:14.078 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:14.078 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:14.078 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:14.078 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:14.078 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:14.079 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:14.079 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:14.079 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:14.079 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.092+0000 7fc32cc1b640 5 monclient: authenticate success, global_id 4292 2024-03-29T10:41:14.079 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.092+0000 7fc32cc1b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:14.079 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.092+0000 7fc3267fc640 1 -- 172.21.15.57:0/3919392863 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc318001c30 con 0x7fc3280ae220 2024-03-29T10:41:14.079 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.092+0000 7fc3267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:14.079 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.092+0000 7fc3267fc640 1 -- 172.21.15.57:0/3919392863 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fc318002040 con 0x7fc3280ae220 2024-03-29T10:41:14.079 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.092+0000 7fc32cc1b640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:14.079 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.092+0000 7fc32cc1b640 4 set_mon_vals no callback set 2024-03-29T10:41:14.080 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.092+0000 7fc3267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:14.080 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.092+0000 7fc3267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:14.080 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.092+0000 7fc3267fc640 10 monclient: dump: 2024-03-29T10:41:14.080 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:14.080 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:14.080 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:14.080 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:14.080 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:14.080 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:14.080 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:14.080 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:14.080 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:14.081 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:14.081 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.093+0000 7fc32cc1b640 10 monclient: shutdown 2024-03-29T10:41:14.081 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.093+0000 7fc32cc1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:14.081 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.093+0000 7fc32cc1b640 1 -- 172.21.15.57:0/3919392863 >> v1:172.21.15.57:6790/0 conn(0x7fc3280ae220 legacy=0x7fc32813b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:14.081 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.093+0000 7fc32cc1b640 1 -- 172.21.15.57:0/3919392863 shutdown_connections 2024-03-29T10:41:14.082 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.093+0000 7fc32cc1b640 1 -- 172.21.15.57:0/3919392863 >> 172.21.15.57:0/3919392863 conn(0x7fc3280673e0 msgr2=0x7fc3280b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:14.082 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.093+0000 7fc32cc1b640 1 -- 172.21.15.57:0/3919392863 shutdown_connections 2024-03-29T10:41:14.082 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.093+0000 7fc32cc1b640 1 -- 172.21.15.57:0/3919392863 wait complete. 2024-03-29T10:41:14.091 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.094+0000 7fc32cc1b640 10 monclient: build_initial_monmap 2024-03-29T10:41:14.091 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.094+0000 7fc32cc1b640 10 monclient: monmap: 2024-03-29T10:41:14.091 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:41:14.091 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:14.091 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:41:14.092 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:41:14.092 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:14.092 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:14.092 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:41:14.092 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:41:14.092 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:41:14.092 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:14.092 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.094+0000 7fc32cc1b640 1 librados: starting msgr at 2024-03-29T10:41:14.092 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.094+0000 7fc32cc1b640 1 librados: starting objecter 2024-03-29T10:41:14.092 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.094+0000 7fc32cc1b640 1 Processor -- start 2024-03-29T10:41:14.092 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.095+0000 7fc32cc1b640 1 -- start start 2024-03-29T10:41:14.092 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.095+0000 7fc32cc1b640 1 librados: setting wanted keys 2024-03-29T10:41:14.092 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.095+0000 7fc32cc1b640 1 librados: calling monclient init 2024-03-29T10:41:14.092 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.095+0000 7fc32cc1b640 10 monclient: init 2024-03-29T10:41:14.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.096+0000 7fc32cc1b640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:14.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.096+0000 7fc32cc1b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:41:14.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.096+0000 7fc32cc1b640 10 monclient(hunting): picked mon.noname-a con 0x7fc3280a9750 addr v1:172.21.15.57:6789/0 2024-03-29T10:41:14.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.096+0000 7fc32cc1b640 10 monclient(hunting): picked mon.noname-b con 0x7fc3280ae220 addr v1:172.21.15.179:6789/0 2024-03-29T10:41:14.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.096+0000 7fc32cc1b640 10 monclient(hunting): picked mon.noname-c con 0x7fc32813b9d0 addr v1:172.21.15.57:6790/0 2024-03-29T10:41:14.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.096+0000 7fc32cc1b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc328132410 con 0x7fc3280a9750 2024-03-29T10:41:14.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.096+0000 7fc32cc1b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc32806e8d0 con 0x7fc3280ae220 2024-03-29T10:41:14.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.096+0000 7fc32cc1b640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc328071c00 con 0x7fc32813b9d0 2024-03-29T10:41:14.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.096+0000 7fc32cc1b640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:14.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.096+0000 7fc32cc1b640 10 monclient(hunting): authenticate will time out at 1026.272297s 2024-03-29T10:41:14.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.098+0000 7fc326ffd640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fc3280ae220 0x7fc328131d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:59468/0 (socket says 172.21.15.57:59468) 2024-03-29T10:41:14.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.098+0000 7fc326ffd640 1 -- 172.21.15.57:0/24022081 learned_addr learned my addr 172.21.15.57:0/24022081 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:41:14.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.098+0000 7fc31ffff640 10 client.?.objecter ms_handle_connect 0x7fc3280ae220 2024-03-29T10:41:14.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.098+0000 7fc31ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:14.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.101+0000 7fc31ffff640 1 -- 172.21.15.57:0/24022081 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920316325 0 0) 0x7fc32806e8d0 con 0x7fc3280ae220 2024-03-29T10:41:14.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.101+0000 7fc31ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:14.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.101+0000 7fc31ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:14.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.101+0000 7fc31ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:14.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.101+0000 7fc31ffff640 10 monclient(hunting): my global_id is 4288 2024-03-29T10:41:14.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.101+0000 7fc31ffff640 1 -- 172.21.15.57:0/24022081 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc308004eb0 con 0x7fc3280ae220 2024-03-29T10:41:14.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.101+0000 7fc31ffff640 10 client.?.objecter ms_handle_connect 0x7fc32813b9d0 2024-03-29T10:41:14.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.101+0000 7fc31ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:14.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.101+0000 7fc31ffff640 1 -- 172.21.15.57:0/24022081 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4125259376 0 0) 0x7fc308004eb0 con 0x7fc3280ae220 2024-03-29T10:41:14.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.101+0000 7fc31ffff640 1 -- 172.21.15.57:0/24022081 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc32806e8d0 con 0x7fc3280ae220 2024-03-29T10:41:14.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.102+0000 7fc31ffff640 1 -- 172.21.15.57:0/24022081 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fc318001f40 con 0x7fc3280ae220 2024-03-29T10:41:14.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.102+0000 7fc31ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:14.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.102+0000 7fc31ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:14.095 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.102+0000 7fc31ffff640 10 monclient(hunting): dump: 2024-03-29T10:41:14.095 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:14.095 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:14.095 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:14.095 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:14.095 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:14.095 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:14.095 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:14.095 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:14.095 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:14.095 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:14.095 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.102+0000 7fc31ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:14.095 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.102+0000 7fc31ffff640 10 client.?.objecter ms_handle_connect 0x7fc3280a9750 2024-03-29T10:41:14.095 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.102+0000 7fc31ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:14.095 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.102+0000 7fc31ffff640 1 -- 172.21.15.57:0/24022081 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693762262 0 0) 0x7fc328071c00 con 0x7fc32813b9d0 2024-03-29T10:41:14.096 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.102+0000 7fc31ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:14.096 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.102+0000 7fc31ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:14.096 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.102+0000 7fc31ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:14.096 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.102+0000 7fc31ffff640 10 monclient(hunting): my global_id is 4295 2024-03-29T10:41:14.096 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.102+0000 7fc31ffff640 1 -- 172.21.15.57:0/24022081 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc308004eb0 con 0x7fc32813b9d0 2024-03-29T10:41:14.096 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.102+0000 7fc31ffff640 1 -- 172.21.15.57:0/24022081 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146366086 0 0) 0x7fc328132410 con 0x7fc3280a9750 2024-03-29T10:41:14.096 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.102+0000 7fc31ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:14.096 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.103+0000 7fc31ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:14.096 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.103+0000 7fc31ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:14.096 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.103+0000 7fc31ffff640 10 monclient(hunting): my global_id is 4278 2024-03-29T10:41:14.096 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.103+0000 7fc31ffff640 1 -- 172.21.15.57:0/24022081 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc328071c00 con 0x7fc3280a9750 2024-03-29T10:41:14.096 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.103+0000 7fc31ffff640 1 -- 172.21.15.57:0/24022081 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 134127464 0 0) 0x7fc32806e8d0 con 0x7fc3280ae220 2024-03-29T10:41:14.096 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.103+0000 7fc31ffff640 1 -- 172.21.15.57:0/24022081 >> v1:172.21.15.57:6790/0 conn(0x7fc32813b9d0 legacy=0x7fc3281eea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:14.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.103+0000 7fc31ffff640 1 -- 172.21.15.57:0/24022081 >> v1:172.21.15.57:6789/0 conn(0x7fc3280a9750 legacy=0x7fc3281315f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:14.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.103+0000 7fc31ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:41:14.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.103+0000 7fc31ffff640 1 monclient: found mon.b 2024-03-29T10:41:14.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.103+0000 7fc31ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:14.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.103+0000 7fc31ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:14.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.103+0000 7fc31ffff640 1 -- 172.21.15.57:0/24022081 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc328071ee0 con 0x7fc3280ae220 2024-03-29T10:41:14.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.103+0000 7fc31ffff640 10 monclient: _finish_auth 0 2024-03-29T10:41:14.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.103+0000 7fc31ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:41:14.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.103+0000 7fc31ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:14.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.103+0000 7fc32cc1b640 5 monclient: authenticate success, global_id 4288 2024-03-29T10:41:14.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.103+0000 7fc32cc1b640 10 monclient: _renew_subs 2024-03-29T10:41:14.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.103+0000 7fc32cc1b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:14.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.103+0000 7fc32cc1b640 1 -- 172.21.15.57:0/24022081 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc32806eb70 con 0x7fc3280ae220 2024-03-29T10:41:14.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.103+0000 7fc32cc1b640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:14.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.103+0000 7fc32cc1b640 10 monclient: _renew_subs 2024-03-29T10:41:14.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.103+0000 7fc32cc1b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:14.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.103+0000 7fc32cc1b640 1 -- 172.21.15.57:0/24022081 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc328070c70 con 0x7fc3280ae220 2024-03-29T10:41:14.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.103+0000 7fc32cc1b640 1 librados: init done 2024-03-29T10:41:14.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.104+0000 7fc31ffff640 1 -- 172.21.15.57:0/24022081 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc318004f40 con 0x7fc3280ae220 2024-03-29T10:41:14.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.104+0000 7fc31ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:14.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.104+0000 7fc31ffff640 1 -- 172.21.15.57:0/24022081 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fc3180052d0 con 0x7fc3280ae220 2024-03-29T10:41:14.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.104+0000 7fc31ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:14.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.104+0000 7fc31ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:14.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.104+0000 7fc31ffff640 10 monclient: dump: 2024-03-29T10:41:14.099 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:14.099 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:14.099 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:14.099 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:14.099 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:14.099 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:14.099 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:14.099 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:14.099 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:14.099 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:14.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.104+0000 7fc3257fa640 4 set_mon_vals no callback set 2024-03-29T10:41:14.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.105+0000 7fc32cc1b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:14.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.105+0000 7fc32cc1b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:14.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.105+0000 7fc32cc1b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:14.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.105+0000 7fc32cc1b640 1 -- 172.21.15.57:0/24022081 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc30c005080 con 0x7fc3280ae220 2024-03-29T10:41:14.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.106+0000 7fc31ffff640 1 -- 172.21.15.57:0/24022081 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fc318005430 con 0x7fc3280ae220 2024-03-29T10:41:14.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.106+0000 7fc31ffff640 1 -- 172.21.15.57:0/24022081 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7fc318084ff0 con 0x7fc3280ae220 2024-03-29T10:41:14.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.106+0000 7fc31ffff640 10 client.4288.objecter ms_dispatch 0x7fc328000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:14.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.106+0000 7fc31ffff640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:14.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.106+0000 7fc31ffff640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:14.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.107+0000 7fc31ffff640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-29T10:41:14.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.107+0000 7fc3277fe640 10 monclient: get_auth_request con 0x7fc30806de50 auth_method 0 2024-03-29T10:41:14.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.107+0000 7fc31ffff640 10 client.4288.objecter ms_handle_connect 0x7fc30806de50 2024-03-29T10:41:14.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.116+0000 7fc31ffff640 1 -- 172.21.15.57:0/24022081 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc318050700 con 0x7fc3280ae220 2024-03-29T10:41:14.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.116+0000 7fc31ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:14.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.116+0000 7fc31ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:14.423 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.440+0000 7fc32cc1b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:14.424 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.440+0000 7fc32cc1b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:14.424 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.440+0000 7fc32cc1b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:14.424 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.440+0000 7fc32cc1b640 1 -- 172.21.15.57:0/24022081 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc30c005330 con 0x7fc3280ae220 2024-03-29T10:41:14.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.443+0000 7fc31ffff640 1 -- 172.21.15.57:0/24022081 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7fc318052800 con 0x7fc3280ae220 2024-03-29T10:41:14.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.443+0000 7fc31ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:14.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.443+0000 7fc31ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:14.428 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:41:14.428 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:14.432 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.448+0000 7fc32cc1b640 10 librados: watch_flush enter 2024-03-29T10:41:14.432 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.448+0000 7fc32cc1b640 10 librados: watch_flush exit 2024-03-29T10:41:14.432 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.448+0000 7fc32cc1b640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-29T10:41:14.432 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.448+0000 7fc32cc1b640 10 client.4288.objecter successfully canceled tick 2024-03-29T10:41:14.432 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.448+0000 7fc32cc1b640 1 -- 172.21.15.57:0/24022081 >> v1:172.21.15.179:6816/2698295936 conn(0x7fc30806de50 legacy=0x7fc308070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:14.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.448+0000 7fc32cc1b640 10 monclient: shutdown 2024-03-29T10:41:14.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.448+0000 7fc32cc1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:14.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.448+0000 7fc32cc1b640 1 -- 172.21.15.57:0/24022081 >> v1:172.21.15.179:6789/0 conn(0x7fc3280ae220 legacy=0x7fc328131d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:14.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.449+0000 7fc32cc1b640 1 -- 172.21.15.57:0/24022081 shutdown_connections 2024-03-29T10:41:14.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.449+0000 7fc32cc1b640 1 -- 172.21.15.57:0/24022081 >> 172.21.15.57:0/24022081 conn(0x7fc3280673e0 msgr2=0x7fc328069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:14.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.449+0000 7fc32cc1b640 1 -- 172.21.15.57:0/24022081 shutdown_connections 2024-03-29T10:41:14.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.449+0000 7fc32cc1b640 1 -- 172.21.15.57:0/24022081 wait complete. 2024-03-29T10:41:14.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:14.449+0000 7fc32cc1b640 1 librados: shutdown 2024-03-29T10:41:14.449 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:21.256 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:21.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.432+0000 7f20c8aa2640 10 monclient: get_monmap_and_config 2024-03-29T10:41:21.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.432+0000 7f20c8aa2640 10 monclient: build_initial_monmap 2024-03-29T10:41:21.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.432+0000 7f20c8aa2640 10 monclient: monmap: 2024-03-29T10:41:21.416 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:41:21.416 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:21.416 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:41:21.433150+0000 2024-03-29T10:41:21.416 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:41:21.433150+0000 2024-03-29T10:41:21.416 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:21.417 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:21.417 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:41:21.417 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:41:21.417 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:41:21.417 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:21.419 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.434+0000 7f20c8aa2640 1 Processor -- start 2024-03-29T10:41:21.419 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.434+0000 7f20c8aa2640 1 -- start start 2024-03-29T10:41:21.419 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.434+0000 7f20c8aa2640 10 monclient: init 2024-03-29T10:41:21.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.435+0000 7f20c8aa2640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:21.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.435+0000 7f20c8aa2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:41:21.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.435+0000 7f20c8aa2640 10 monclient(hunting): picked mon.noname-c con 0x7f20bc034850 addr v1:172.21.15.179:6789/0 2024-03-29T10:41:21.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.435+0000 7f20c8aa2640 10 monclient(hunting): picked mon.noname-b con 0x7f20bc02a640 addr v1:172.21.15.57:6790/0 2024-03-29T10:41:21.421 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.435+0000 7f20c8aa2640 10 monclient(hunting): picked mon.noname-a con 0x7f20bc0327d0 addr v1:172.21.15.57:6789/0 2024-03-29T10:41:21.422 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.438+0000 7f20c8aa2640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20bc02c820 con 0x7f20bc0327d0 2024-03-29T10:41:21.422 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.438+0000 7f20c8aa2640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20bc02a120 con 0x7f20bc034850 2024-03-29T10:41:21.422 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.438+0000 7f20c8aa2640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20bc0c75b0 con 0x7f20bc02a640 2024-03-29T10:41:21.422 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.438+0000 7f20c8aa2640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:21.422 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.438+0000 7f20c8aa2640 10 monclient(hunting): authenticate will time out at 1033.614727s 2024-03-29T10:41:21.422 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.438+0000 7f20c3fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f20bc0327d0 0x7f20bc0cf240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:53623/0 (socket says 172.21.15.57:53623) 2024-03-29T10:41:21.422 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.438+0000 7f20c3fff640 1 -- 172.21.15.57:0/3847913267 learned_addr learned my addr 172.21.15.57:0/3847913267 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:41:21.425 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.440+0000 7f20c27fc640 1 -- 172.21.15.57:0/3847913267 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089961786 0 0) 0x7f20bc02c820 con 0x7f20bc0327d0 2024-03-29T10:41:21.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.441+0000 7f20c27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:21.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.441+0000 7f20c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:21.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.441+0000 7f20c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:21.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.441+0000 7f20c27fc640 10 monclient(hunting): my global_id is 4281 2024-03-29T10:41:21.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.441+0000 7f20c27fc640 1 -- 172.21.15.57:0/3847913267 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20b4003520 con 0x7f20bc0327d0 2024-03-29T10:41:21.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.441+0000 7f20c27fc640 1 -- 172.21.15.57:0/3847913267 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3640947406 0 0) 0x7f20b4003520 con 0x7f20bc0327d0 2024-03-29T10:41:21.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.441+0000 7f20c27fc640 1 -- 172.21.15.57:0/3847913267 >> v1:172.21.15.57:6790/0 conn(0x7f20bc02a640 legacy=0x7f20bc0320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:41:21.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.441+0000 7f20c27fc640 1 -- 172.21.15.57:0/3847913267 >> v1:172.21.15.179:6789/0 conn(0x7f20bc034850 legacy=0x7f20bc0c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:41:21.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.441+0000 7f20c27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:41:21.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.441+0000 7f20c27fc640 1 monclient: found mon.noname-a 2024-03-29T10:41:21.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.441+0000 7f20c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:21.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.442+0000 7f20c27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:41:21.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.442+0000 7f20c27fc640 1 -- 172.21.15.57:0/3847913267 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20bc033110 con 0x7f20bc0327d0 2024-03-29T10:41:21.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.442+0000 7f20c27fc640 10 monclient: _finish_auth 0 2024-03-29T10:41:21.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.442+0000 7f20c27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:41:21.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.442+0000 7f20c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:21.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.442+0000 7f20c27fc640 1 -- 172.21.15.57:0/3847913267 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f20c404f730 con 0x7f20bc0327d0 2024-03-29T10:41:21.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.442+0000 7f20c8aa2640 5 monclient: authenticate success, global_id 4281 2024-03-29T10:41:21.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.442+0000 7f20c8aa2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:21.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.442+0000 7f20c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:21.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.442+0000 7f20c27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:21.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.442+0000 7f20c27fc640 10 monclient: dump: 2024-03-29T10:41:21.427 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:21.427 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:21.427 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:21.427 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:21.427 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:21.427 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:21.427 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:21.428 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:21.428 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:21.428 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:21.428 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.442+0000 7f20c8aa2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:21.429 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.444+0000 7f20c27fc640 1 -- 172.21.15.57:0/3847913267 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20c40008d0 con 0x7f20bc0327d0 2024-03-29T10:41:21.429 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.444+0000 7f20c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:21.429 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.444+0000 7f20c27fc640 1 -- 172.21.15.57:0/3847913267 <== mon.0 v1:172.21.15.57:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f20c40758b0 con 0x7f20bc0327d0 2024-03-29T10:41:21.429 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.444+0000 7f20c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:21.429 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.444+0000 7f20c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:21.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.444+0000 7f20c27fc640 10 monclient: dump: 2024-03-29T10:41:21.430 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:21.430 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:21.430 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:21.430 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:21.430 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:21.430 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:21.430 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:21.430 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:21.430 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:21.430 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:21.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.444+0000 7f20c8aa2640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:21.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.444+0000 7f20c8aa2640 4 set_mon_vals no callback set 2024-03-29T10:41:21.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.445+0000 7f20c8aa2640 10 monclient: shutdown 2024-03-29T10:41:21.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.445+0000 7f20c8aa2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:21.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.445+0000 7f20c8aa2640 1 -- 172.21.15.57:0/3847913267 >> v1:172.21.15.57:6789/0 conn(0x7f20bc0327d0 legacy=0x7f20bc0cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:21.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.445+0000 7f20c8aa2640 1 -- 172.21.15.57:0/3847913267 shutdown_connections 2024-03-29T10:41:21.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.446+0000 7f20c8aa2640 1 -- 172.21.15.57:0/3847913267 >> 172.21.15.57:0/3847913267 conn(0x7f20bc0058f0 msgr2=0x7f20bc005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:21.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.448+0000 7f20c8aa2640 1 -- 172.21.15.57:0/3847913267 shutdown_connections 2024-03-29T10:41:21.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.448+0000 7f20c8aa2640 1 -- 172.21.15.57:0/3847913267 wait complete. 2024-03-29T10:41:21.432 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.449+0000 7f20c8aa2640 10 monclient: build_initial_monmap 2024-03-29T10:41:21.432 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.449+0000 7f20c8aa2640 10 monclient: monmap: 2024-03-29T10:41:21.432 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:41:21.432 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:21.432 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:41:21.432 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:41:21.432 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:21.432 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:21.432 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:41:21.432 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:41:21.432 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:41:21.433 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:21.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.449+0000 7f20c8aa2640 1 librados: starting msgr at 2024-03-29T10:41:21.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.449+0000 7f20c8aa2640 1 librados: starting objecter 2024-03-29T10:41:21.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.449+0000 7f20c8aa2640 1 Processor -- start 2024-03-29T10:41:21.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.450+0000 7f20c8aa2640 1 -- start start 2024-03-29T10:41:21.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.450+0000 7f20c8aa2640 1 librados: setting wanted keys 2024-03-29T10:41:21.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.450+0000 7f20c8aa2640 1 librados: calling monclient init 2024-03-29T10:41:21.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.450+0000 7f20c8aa2640 10 monclient: init 2024-03-29T10:41:21.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.451+0000 7f20c8aa2640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:21.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.451+0000 7f20c8aa2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:41:21.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.451+0000 7f20c8aa2640 10 monclient(hunting): picked mon.noname-b con 0x7f20bc02a640 addr v1:172.21.15.179:6789/0 2024-03-29T10:41:21.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.451+0000 7f20c8aa2640 10 monclient(hunting): picked mon.noname-c con 0x7f20bc0327d0 addr v1:172.21.15.57:6790/0 2024-03-29T10:41:21.440 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.454+0000 7f20c8aa2640 10 monclient(hunting): picked mon.noname-a con 0x7f20bc034850 addr v1:172.21.15.57:6789/0 2024-03-29T10:41:21.440 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.455+0000 7f20c8aa2640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20bc0bcd90 con 0x7f20bc034850 2024-03-29T10:41:21.440 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.455+0000 7f20c8aa2640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20bc0bcfc0 con 0x7f20bc02a640 2024-03-29T10:41:21.440 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.455+0000 7f20c8aa2640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20bc0bd150 con 0x7f20bc0327d0 2024-03-29T10:41:21.440 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.455+0000 7f20c8aa2640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:21.440 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.455+0000 7f20c8aa2640 10 monclient(hunting): authenticate will time out at 1033.631633s 2024-03-29T10:41:21.440 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.455+0000 7f20c37fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f20bc02a640 0x7f20bc0bc680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:54491/0 (socket says 172.21.15.57:54491) 2024-03-29T10:41:21.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.455+0000 7f20c37fe640 1 -- 172.21.15.57:0/1290295501 learned_addr learned my addr 172.21.15.57:0/1290295501 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:41:21.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.455+0000 7f20c2ffd640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f20bc0327d0 0x7f20bc157010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:55061/0 (socket says 172.21.15.57:55061) 2024-03-29T10:41:21.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.456+0000 7f20affff640 10 client.?.objecter ms_handle_connect 0x7f20bc0327d0 2024-03-29T10:41:21.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.456+0000 7f20affff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:21.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.456+0000 7f20affff640 10 client.?.objecter ms_handle_connect 0x7f20bc02a640 2024-03-29T10:41:21.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.456+0000 7f20affff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:21.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.456+0000 7f20affff640 1 -- 172.21.15.57:0/1290295501 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2346899694 0 0) 0x7f20bc0bcfc0 con 0x7f20bc02a640 2024-03-29T10:41:21.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.456+0000 7f20affff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:21.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.456+0000 7f20affff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:21.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.456+0000 7f20affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:21.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.456+0000 7f20affff640 10 monclient(hunting): my global_id is 4291 2024-03-29T10:41:21.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.456+0000 7f20affff640 1 -- 172.21.15.57:0/1290295501 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20a8004eb0 con 0x7f20bc02a640 2024-03-29T10:41:21.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.456+0000 7f20affff640 1 -- 172.21.15.57:0/1290295501 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860508942 0 0) 0x7f20bc0bd150 con 0x7f20bc0327d0 2024-03-29T10:41:21.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.456+0000 7f20affff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:21.442 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.456+0000 7f20affff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:21.442 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.456+0000 7f20affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:21.442 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.456+0000 7f20affff640 10 monclient(hunting): my global_id is 4298 2024-03-29T10:41:21.442 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.456+0000 7f20affff640 1 -- 172.21.15.57:0/1290295501 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20bc0bcfc0 con 0x7f20bc0327d0 2024-03-29T10:41:21.446 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.458+0000 7f20affff640 1 -- 172.21.15.57:0/1290295501 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 161153018 0 0) 0x7f20a8004eb0 con 0x7f20bc02a640 2024-03-29T10:41:21.446 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.458+0000 7f20affff640 1 -- 172.21.15.57:0/1290295501 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20a8005490 con 0x7f20bc02a640 2024-03-29T10:41:21.446 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.458+0000 7f20affff640 1 -- 172.21.15.57:0/1290295501 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f20b00017d0 con 0x7f20bc02a640 2024-03-29T10:41:21.446 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.458+0000 7f20affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:21.446 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.458+0000 7f20affff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:21.446 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.458+0000 7f20affff640 10 monclient(hunting): dump: 2024-03-29T10:41:21.446 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:21.446 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:21.447 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:21.447 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:21.447 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:21.447 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:21.447 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:21.447 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:21.447 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:21.447 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:21.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.458+0000 7f20affff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:21.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.459+0000 7f20affff640 1 -- 172.21.15.57:0/1290295501 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2481622102 0 0) 0x7f20bc0bcfc0 con 0x7f20bc0327d0 2024-03-29T10:41:21.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.459+0000 7f20affff640 1 -- 172.21.15.57:0/1290295501 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20a8004eb0 con 0x7f20bc0327d0 2024-03-29T10:41:21.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.459+0000 7f20affff640 1 -- 172.21.15.57:0/1290295501 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f20b8003f40 con 0x7f20bc0327d0 2024-03-29T10:41:21.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.459+0000 7f20affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:21.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.459+0000 7f20affff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:21.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.459+0000 7f20affff640 10 monclient(hunting): dump: 2024-03-29T10:41:21.448 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:21.448 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:21.448 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:21.448 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:21.448 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:21.448 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:21.448 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:21.448 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:21.448 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:21.448 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:21.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.459+0000 7f20affff640 1 -- 172.21.15.57:0/1290295501 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2567158417 0 0) 0x7f20a8005490 con 0x7f20bc02a640 2024-03-29T10:41:21.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.459+0000 7f20affff640 1 -- 172.21.15.57:0/1290295501 >> v1:172.21.15.57:6790/0 conn(0x7f20bc0327d0 legacy=0x7f20bc157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:21.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.459+0000 7f20affff640 1 -- 172.21.15.57:0/1290295501 >> v1:172.21.15.57:6789/0 conn(0x7f20bc034850 legacy=0x7f20bc157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:21.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.459+0000 7f20affff640 10 monclient: _finish_hunting 0 2024-03-29T10:41:21.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.459+0000 7f20affff640 1 monclient: found mon.b 2024-03-29T10:41:21.449 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.459+0000 7f20affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:21.449 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.459+0000 7f20affff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:21.449 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.459+0000 7f20affff640 1 -- 172.21.15.57:0/1290295501 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20bc15a580 con 0x7f20bc02a640 2024-03-29T10:41:21.449 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.459+0000 7f20affff640 10 monclient: _finish_auth 0 2024-03-29T10:41:21.449 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.459+0000 7f20affff640 10 monclient: _check_auth_tickets 2024-03-29T10:41:21.449 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.459+0000 7f20affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:21.449 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.459+0000 7f20affff640 1 -- 172.21.15.57:0/1290295501 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20b0001ce0 con 0x7f20bc02a640 2024-03-29T10:41:21.449 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.459+0000 7f20affff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:21.449 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.459+0000 7f20affff640 1 -- 172.21.15.57:0/1290295501 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f20b0004a00 con 0x7f20bc02a640 2024-03-29T10:41:21.449 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.459+0000 7f20affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:21.449 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.459+0000 7f20affff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:21.449 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.459+0000 7f20affff640 10 monclient: dump: 2024-03-29T10:41:21.449 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:21.449 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:21.449 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:21.450 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:21.450 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:21.450 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:21.450 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:21.450 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:21.450 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:21.450 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:21.450 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.460+0000 7f20c1ffb640 4 set_mon_vals no callback set 2024-03-29T10:41:21.450 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.460+0000 7f20c8aa2640 5 monclient: authenticate success, global_id 4291 2024-03-29T10:41:21.450 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.460+0000 7f20c8aa2640 10 monclient: _renew_subs 2024-03-29T10:41:21.450 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.460+0000 7f20c8aa2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:21.450 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.460+0000 7f20c8aa2640 1 -- 172.21.15.57:0/1290295501 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20bc15a770 con 0x7f20bc02a640 2024-03-29T10:41:21.450 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.460+0000 7f20c8aa2640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:21.450 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.460+0000 7f20c8aa2640 10 monclient: _renew_subs 2024-03-29T10:41:21.451 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.460+0000 7f20c8aa2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:21.451 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.460+0000 7f20c8aa2640 1 -- 172.21.15.57:0/1290295501 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20bc158010 con 0x7f20bc02a640 2024-03-29T10:41:21.451 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.460+0000 7f20c8aa2640 1 librados: init done 2024-03-29T10:41:21.451 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.461+0000 7f20adffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:21.451 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.462+0000 7f20adffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:21.451 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.462+0000 7f20adffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:21.451 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.462+0000 7f20adffb640 1 -- 172.21.15.57:0/1290295501 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20bc004cd0 con 0x7f20bc02a640 2024-03-29T10:41:21.452 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.464+0000 7f20affff640 1 -- 172.21.15.57:0/1290295501 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f20b001a9d0 con 0x7f20bc02a640 2024-03-29T10:41:21.452 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.464+0000 7f20affff640 1 -- 172.21.15.57:0/1290295501 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f20b0084910 con 0x7f20bc02a640 2024-03-29T10:41:21.452 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.464+0000 7f20affff640 10 client.4291.objecter ms_dispatch 0x7f20c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:21.452 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.464+0000 7f20affff640 3 client.4291.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:21.452 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.464+0000 7f20affff640 3 client.4291.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:21.452 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.464+0000 7f20affff640 20 client.4291.objecter dump_active .. 0 homeless 2024-03-29T10:41:21.452 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.465+0000 7f20c2ffd640 10 monclient: get_auth_request con 0x7f20a8072350 auth_method 0 2024-03-29T10:41:21.452 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.467+0000 7f20affff640 10 client.4291.objecter ms_handle_connect 0x7f20a8072350 2024-03-29T10:41:21.455 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.470+0000 7f20affff640 1 -- 172.21.15.57:0/1290295501 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20b0050020 con 0x7f20bc02a640 2024-03-29T10:41:21.456 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.470+0000 7f20affff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:21.456 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.470+0000 7f20affff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:21.784 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.801+0000 7f20adffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:21.784 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.801+0000 7f20adffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:21.785 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.801+0000 7f20adffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:21.785 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.801+0000 7f20adffb640 1 -- 172.21.15.57:0/1290295501 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20bc0086f0 con 0x7f20bc02a640 2024-03-29T10:41:21.788 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.805+0000 7f20affff640 1 -- 172.21.15.57:0/1290295501 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f20b0052120 con 0x7f20bc02a640 2024-03-29T10:41:21.788 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.805+0000 7f20affff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:21.788 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.805+0000 7f20affff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:21.789 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:41:21.789 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:21.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.810+0000 7f20adffb640 10 librados: watch_flush enter 2024-03-29T10:41:21.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.810+0000 7f20adffb640 10 librados: watch_flush exit 2024-03-29T10:41:21.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.810+0000 7f20adffb640 20 client.4291.objecter shutdown clearing up homeless session... 2024-03-29T10:41:21.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.810+0000 7f20adffb640 10 client.4291.objecter successfully canceled tick 2024-03-29T10:41:21.794 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.811+0000 7f20adffb640 1 -- 172.21.15.57:0/1290295501 >> v1:172.21.15.179:6816/2698295936 conn(0x7f20a8072350 legacy=0x7f20a8074810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:21.794 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.811+0000 7f20adffb640 10 monclient: shutdown 2024-03-29T10:41:21.794 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.811+0000 7f20adffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:21.794 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.811+0000 7f20adffb640 1 -- 172.21.15.57:0/1290295501 >> v1:172.21.15.179:6789/0 conn(0x7f20bc02a640 legacy=0x7f20bc0bc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:21.795 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.811+0000 7f20adffb640 1 -- 172.21.15.57:0/1290295501 shutdown_connections 2024-03-29T10:41:21.795 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.811+0000 7f20adffb640 1 -- 172.21.15.57:0/1290295501 >> 172.21.15.57:0/1290295501 conn(0x7f20bc0058f0 msgr2=0x7f20bc0372c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:21.795 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.812+0000 7f20adffb640 1 -- 172.21.15.57:0/1290295501 shutdown_connections 2024-03-29T10:41:21.796 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.812+0000 7f20adffb640 1 -- 172.21.15.57:0/1290295501 wait complete. 2024-03-29T10:41:21.796 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:21.813+0000 7f20adffb640 1 librados: shutdown 2024-03-29T10:41:21.814 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:28.621 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:28.772 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.788+0000 7f06a7833640 10 monclient: get_monmap_and_config 2024-03-29T10:41:28.772 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.788+0000 7f06a7833640 10 monclient: build_initial_monmap 2024-03-29T10:41:28.772 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.788+0000 7f06a7833640 10 monclient: monmap: 2024-03-29T10:41:28.772 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:41:28.772 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:28.772 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:41:28.788971+0000 2024-03-29T10:41:28.772 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:41:28.788971+0000 2024-03-29T10:41:28.773 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:28.773 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:28.773 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:41:28.773 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:41:28.773 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:41:28.773 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:28.775 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.789+0000 7f06a7833640 1 Processor -- start 2024-03-29T10:41:28.775 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.790+0000 7f06a7833640 1 -- start start 2024-03-29T10:41:28.775 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.790+0000 7f06a7833640 10 monclient: init 2024-03-29T10:41:28.775 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.790+0000 7f06a7833640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:28.775 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.790+0000 7f06a7833640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:41:28.775 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.790+0000 7f06a7833640 10 monclient(hunting): picked mon.noname-c con 0x7f06a00a9ac0 addr v1:172.21.15.179:6789/0 2024-03-29T10:41:28.779 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.790+0000 7f06a7833640 10 monclient(hunting): picked mon.noname-b con 0x7f06a013ee90 addr v1:172.21.15.57:6790/0 2024-03-29T10:41:28.779 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.790+0000 7f06a7833640 10 monclient(hunting): picked mon.noname-a con 0x7f06a00af900 addr v1:172.21.15.57:6789/0 2024-03-29T10:41:28.779 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.790+0000 7f06a7833640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a00afd10 con 0x7f06a00af900 2024-03-29T10:41:28.779 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.790+0000 7f06a7833640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a00af6f0 con 0x7f06a00a9ac0 2024-03-29T10:41:28.779 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.790+0000 7f06a7833640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a00aac80 con 0x7f06a013ee90 2024-03-29T10:41:28.780 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.791+0000 7f06a7833640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:28.780 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.791+0000 7f06a7833640 10 monclient(hunting): authenticate will time out at 1040.967034s 2024-03-29T10:41:28.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.796+0000 7f06a4dce640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f06a013ee90 0x7f06a013b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:45791/0 (socket says 172.21.15.57:45791) 2024-03-29T10:41:28.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.796+0000 7f06a4dce640 1 -- 172.21.15.57:0/786902459 learned_addr learned my addr 172.21.15.57:0/786902459 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:41:28.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.796+0000 7f06a55cf640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f06a00a9ac0 0x7f06a00adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:43737/0 (socket says 172.21.15.57:43737) 2024-03-29T10:41:28.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.797+0000 7f0697fff640 1 -- 172.21.15.57:0/786902459 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943242936 0 0) 0x7f06a00af6f0 con 0x7f06a00a9ac0 2024-03-29T10:41:28.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.798+0000 7f0697fff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:28.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.798+0000 7f0697fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:28.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.798+0000 7f0697fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:28.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.798+0000 7f0697fff640 10 monclient(hunting): my global_id is 4294 2024-03-29T10:41:28.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.798+0000 7f0697fff640 1 -- 172.21.15.57:0/786902459 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f068c003520 con 0x7f06a00a9ac0 2024-03-29T10:41:28.784 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 1 -- 172.21.15.57:0/786902459 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157370909 0 0) 0x7f06a00afd10 con 0x7f06a00af900 2024-03-29T10:41:28.785 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:28.785 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:28.785 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:28.785 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 10 monclient(hunting): my global_id is 4284 2024-03-29T10:41:28.785 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 1 -- 172.21.15.57:0/786902459 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06a00af6f0 con 0x7f06a00af900 2024-03-29T10:41:28.785 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 1 -- 172.21.15.57:0/786902459 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496471645 0 0) 0x7f06a00aac80 con 0x7f06a013ee90 2024-03-29T10:41:28.785 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:28.785 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:28.785 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:28.785 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 10 monclient(hunting): my global_id is 4301 2024-03-29T10:41:28.785 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 1 -- 172.21.15.57:0/786902459 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06a00afd10 con 0x7f06a013ee90 2024-03-29T10:41:28.785 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 1 -- 172.21.15.57:0/786902459 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3777910881 0 0) 0x7f06a00afd10 con 0x7f06a013ee90 2024-03-29T10:41:28.785 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 1 -- 172.21.15.57:0/786902459 >> v1:172.21.15.179:6789/0 conn(0x7f06a00a9ac0 legacy=0x7f06a00adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:28.785 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 1 -- 172.21.15.57:0/786902459 >> v1:172.21.15.57:6789/0 conn(0x7f06a00af900 legacy=0x7f06a0149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:28.786 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 10 monclient: _finish_hunting 0 2024-03-29T10:41:28.786 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 1 monclient: found mon.noname-b 2024-03-29T10:41:28.786 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:28.786 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:41:28.786 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 1 -- 172.21.15.57:0/786902459 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a00aaf60 con 0x7f06a013ee90 2024-03-29T10:41:28.786 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 10 monclient: _finish_auth 0 2024-03-29T10:41:28.786 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 10 monclient: _check_auth_tickets 2024-03-29T10:41:28.786 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:28.786 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 1 -- 172.21.15.57:0/786902459 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f0690004e00 con 0x7f06a013ee90 2024-03-29T10:41:28.786 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:28.786 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:28.787 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 10 monclient: dump: 2024-03-29T10:41:28.787 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:28.787 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:28.787 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:28.787 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:28.787 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:28.787 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:28.787 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:28.787 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:28.787 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:28.787 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:28.787 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 1 -- 172.21.15.57:0/786902459 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0690000ed0 con 0x7f06a013ee90 2024-03-29T10:41:28.787 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:28.787 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 1 -- 172.21.15.57:0/786902459 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f0690004e00 con 0x7f06a013ee90 2024-03-29T10:41:28.788 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:28.788 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:28.788 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f0697fff640 10 monclient: dump: 2024-03-29T10:41:28.788 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:28.788 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:28.788 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:28.788 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:28.788 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:28.788 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:28.788 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:28.788 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:28.788 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:28.788 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:28.789 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f06a7833640 5 monclient: authenticate success, global_id 4301 2024-03-29T10:41:28.789 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.799+0000 7f06a7833640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:28.789 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.800+0000 7f06a7833640 4 set_mon_vals no callback set 2024-03-29T10:41:28.789 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.800+0000 7f06a7833640 10 monclient: shutdown 2024-03-29T10:41:28.789 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.800+0000 7f06a7833640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:28.789 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.800+0000 7f06a7833640 1 -- 172.21.15.57:0/786902459 >> v1:172.21.15.57:6790/0 conn(0x7f06a013ee90 legacy=0x7f06a013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:28.789 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.801+0000 7f06a7833640 1 -- 172.21.15.57:0/786902459 shutdown_connections 2024-03-29T10:41:28.789 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.801+0000 7f06a7833640 1 -- 172.21.15.57:0/786902459 >> 172.21.15.57:0/786902459 conn(0x7f06a00b35e0 msgr2=0x7f06a00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:28.791 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.803+0000 7f06a7833640 1 -- 172.21.15.57:0/786902459 shutdown_connections 2024-03-29T10:41:28.791 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.804+0000 7f06a7833640 1 -- 172.21.15.57:0/786902459 wait complete. 2024-03-29T10:41:28.791 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.805+0000 7f06a7833640 10 monclient: build_initial_monmap 2024-03-29T10:41:28.791 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.805+0000 7f06a7833640 10 monclient: monmap: 2024-03-29T10:41:28.791 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:41:28.792 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:28.792 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:41:28.792 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:41:28.792 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:28.792 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:28.792 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:41:28.792 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:41:28.792 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:41:28.792 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:28.792 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.806+0000 7f06a7833640 1 librados: starting msgr at 2024-03-29T10:41:28.792 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.806+0000 7f06a7833640 1 librados: starting objecter 2024-03-29T10:41:28.792 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.806+0000 7f06a7833640 1 Processor -- start 2024-03-29T10:41:28.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.806+0000 7f06a7833640 1 -- start start 2024-03-29T10:41:28.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.806+0000 7f06a7833640 1 librados: setting wanted keys 2024-03-29T10:41:28.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.806+0000 7f06a7833640 1 librados: calling monclient init 2024-03-29T10:41:28.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.806+0000 7f06a7833640 10 monclient: init 2024-03-29T10:41:28.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.807+0000 7f06a7833640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:28.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.807+0000 7f06a7833640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:41:28.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.807+0000 7f06a7833640 10 monclient(hunting): picked mon.noname-a con 0x7f06a00af900 addr v1:172.21.15.57:6789/0 2024-03-29T10:41:28.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.807+0000 7f06a7833640 10 monclient(hunting): picked mon.noname-b con 0x7f06a013ee90 addr v1:172.21.15.179:6789/0 2024-03-29T10:41:28.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.807+0000 7f06a7833640 10 monclient(hunting): picked mon.noname-c con 0x7f06a00706c0 addr v1:172.21.15.57:6790/0 2024-03-29T10:41:28.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.807+0000 7f06a7833640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a0071490 con 0x7f06a00af900 2024-03-29T10:41:28.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.807+0000 7f06a7833640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a0071770 con 0x7f06a013ee90 2024-03-29T10:41:28.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.807+0000 7f06a7833640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a01c6a40 con 0x7f06a00706c0 2024-03-29T10:41:28.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.807+0000 7f06a7833640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:28.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.807+0000 7f06a7833640 10 monclient(hunting): authenticate will time out at 1040.983460s 2024-03-29T10:41:28.795 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.811+0000 7f06a55cf640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f06a00af900 0x7f06a0071f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:58673/0 (socket says 172.21.15.57:58673) 2024-03-29T10:41:28.795 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.811+0000 7f06a55cf640 1 -- 172.21.15.57:0/716901229 learned_addr learned my addr 172.21.15.57:0/716901229 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:41:28.795 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.811+0000 7f0695ffb640 10 client.?.objecter ms_handle_connect 0x7f06a00af900 2024-03-29T10:41:28.795 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.811+0000 7f0695ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:28.795 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.811+0000 7f0695ffb640 1 -- 172.21.15.57:0/716901229 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4240516951 0 0) 0x7f06a0071490 con 0x7f06a00af900 2024-03-29T10:41:28.795 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.811+0000 7f0695ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:41:28.795 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.811+0000 7f0695ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:28.795 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.811+0000 7f0695ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:28.795 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.811+0000 7f0695ffb640 10 monclient(hunting): my global_id is 4287 2024-03-29T10:41:28.795 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.811+0000 7f0695ffb640 1 -- 172.21.15.57:0/716901229 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0684004eb0 con 0x7f06a00af900 2024-03-29T10:41:28.803 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.815+0000 7f0695ffb640 10 client.?.objecter ms_handle_connect 0x7f06a013ee90 2024-03-29T10:41:28.803 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.816+0000 7f0695ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:28.803 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.816+0000 7f0695ffb640 10 client.?.objecter ms_handle_connect 0x7f06a00706c0 2024-03-29T10:41:28.803 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.816+0000 7f0695ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:28.803 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.816+0000 7f0695ffb640 1 -- 172.21.15.57:0/716901229 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3164719731 0 0) 0x7f0684004eb0 con 0x7f06a00af900 2024-03-29T10:41:28.804 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.816+0000 7f0695ffb640 1 -- 172.21.15.57:0/716901229 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06a0071490 con 0x7f06a00af900 2024-03-29T10:41:28.804 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.816+0000 7f0695ffb640 1 -- 172.21.15.57:0/716901229 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419095093 0 0) 0x7f06a01c6a40 con 0x7f06a00706c0 2024-03-29T10:41:28.804 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.816+0000 7f0695ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:41:28.804 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.816+0000 7f0695ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:28.804 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.816+0000 7f0695ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:28.804 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.816+0000 7f0695ffb640 10 monclient(hunting): my global_id is 4304 2024-03-29T10:41:28.804 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.816+0000 7f0695ffb640 1 -- 172.21.15.57:0/716901229 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0684004eb0 con 0x7f06a00706c0 2024-03-29T10:41:28.804 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.816+0000 7f0695ffb640 1 -- 172.21.15.57:0/716901229 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f0698001cd0 con 0x7f06a00af900 2024-03-29T10:41:28.804 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.816+0000 7f0695ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:28.804 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.816+0000 7f0695ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:28.804 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.817+0000 7f0695ffb640 10 monclient(hunting): dump: 2024-03-29T10:41:28.804 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:28.804 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:28.805 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:28.805 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:28.805 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:28.805 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:28.805 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:28.805 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:28.805 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:28.805 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:28.805 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.817+0000 7f0695ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:28.805 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.817+0000 7f0695ffb640 1 -- 172.21.15.57:0/716901229 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3218560463 0 0) 0x7f06a0071770 con 0x7f06a013ee90 2024-03-29T10:41:28.806 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.817+0000 7f0695ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:41:28.806 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.817+0000 7f0695ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:28.806 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.817+0000 7f0695ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:28.806 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.817+0000 7f0695ffb640 10 monclient(hunting): my global_id is 4297 2024-03-29T10:41:28.806 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.817+0000 7f0695ffb640 1 -- 172.21.15.57:0/716901229 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06a01c6a40 con 0x7f06a013ee90 2024-03-29T10:41:28.806 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.817+0000 7f0695ffb640 1 -- 172.21.15.57:0/716901229 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2117687102 0 0) 0x7f06a0071490 con 0x7f06a00af900 2024-03-29T10:41:28.806 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.817+0000 7f0695ffb640 1 -- 172.21.15.57:0/716901229 >> v1:172.21.15.57:6790/0 conn(0x7f06a00706c0 legacy=0x7f06a01e64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:28.806 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.817+0000 7f0695ffb640 1 -- 172.21.15.57:0/716901229 >> v1:172.21.15.179:6789/0 conn(0x7f06a013ee90 legacy=0x7f06a006ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:28.806 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.817+0000 7f0695ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:41:28.807 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.817+0000 7f0695ffb640 1 monclient: found mon.a 2024-03-29T10:41:28.807 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.817+0000 7f0695ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:28.807 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.817+0000 7f0695ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:41:28.807 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.817+0000 7f0695ffb640 1 -- 172.21.15.57:0/716901229 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a006f660 con 0x7f06a00af900 2024-03-29T10:41:28.807 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.818+0000 7f0695ffb640 10 monclient: _finish_auth 0 2024-03-29T10:41:28.807 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.818+0000 7f0695ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:41:28.807 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.818+0000 7f0695ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:28.807 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.818+0000 7f0695ffb640 1 -- 172.21.15.57:0/716901229 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0698001e30 con 0x7f06a00af900 2024-03-29T10:41:28.807 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.818+0000 7f0695ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:28.807 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.818+0000 7f0695ffb640 1 -- 172.21.15.57:0/716901229 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f0698005f20 con 0x7f06a00af900 2024-03-29T10:41:28.807 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.818+0000 7f0695ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:28.807 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.818+0000 7f0695ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:28.808 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.818+0000 7f0695ffb640 10 monclient: dump: 2024-03-29T10:41:28.808 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:28.808 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:28.808 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:28.808 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:28.808 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:28.808 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:28.808 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:28.808 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:28.808 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:28.809 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:28.809 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.818+0000 7f06977fe640 4 set_mon_vals no callback set 2024-03-29T10:41:28.809 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.818+0000 7f06a7833640 5 monclient: authenticate success, global_id 4287 2024-03-29T10:41:28.809 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.818+0000 7f06a7833640 10 monclient: _renew_subs 2024-03-29T10:41:28.809 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.818+0000 7f06a7833640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:41:28.809 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.818+0000 7f06a7833640 1 -- 172.21.15.57:0/716901229 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06a006f8b0 con 0x7f06a00af900 2024-03-29T10:41:28.809 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.819+0000 7f06a7833640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:28.809 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.819+0000 7f06a7833640 10 monclient: _renew_subs 2024-03-29T10:41:28.809 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.819+0000 7f06a7833640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:41:28.809 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.819+0000 7f06a7833640 1 -- 172.21.15.57:0/716901229 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a01e7df0 con 0x7f06a00af900 2024-03-29T10:41:28.809 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.819+0000 7f06a7833640 1 librados: init done 2024-03-29T10:41:28.810 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.822+0000 7f0695ffb640 1 -- 172.21.15.57:0/716901229 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f0698006080 con 0x7f06a00af900 2024-03-29T10:41:28.811 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.824+0000 7f06a7833640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:28.811 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.824+0000 7f06a7833640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:28.811 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.824+0000 7f06a7833640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:41:28.811 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.824+0000 7f06a7833640 1 -- 172.21.15.57:0/716901229 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06a006fc50 con 0x7f06a00af900 2024-03-29T10:41:28.811 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.824+0000 7f0695ffb640 1 -- 172.21.15.57:0/716901229 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f0698084d80 con 0x7f06a00af900 2024-03-29T10:41:28.811 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.824+0000 7f0695ffb640 10 client.4287.objecter ms_dispatch 0x7f06a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:28.811 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.824+0000 7f0695ffb640 3 client.4287.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:28.811 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.824+0000 7f0695ffb640 3 client.4287.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:28.811 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.824+0000 7f0695ffb640 20 client.4287.objecter dump_active .. 0 homeless 2024-03-29T10:41:28.811 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.824+0000 7f06a4dce640 10 monclient: get_auth_request con 0x7f068406dee0 auth_method 0 2024-03-29T10:41:28.811 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.825+0000 7f0695ffb640 10 client.4287.objecter ms_handle_connect 0x7f068406dee0 2024-03-29T10:41:28.819 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.836+0000 7f0695ffb640 1 -- 172.21.15.57:0/716901229 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0698050660 con 0x7f06a00af900 2024-03-29T10:41:28.819 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.836+0000 7f0695ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:28.819 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:28.836+0000 7f0695ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:29.159 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:29.176+0000 7f06a7833640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:29.159 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:29.176+0000 7f06a7833640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:29.160 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:29.176+0000 7f06a7833640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:41:29.160 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:29.176+0000 7f06a7833640 1 -- 172.21.15.57:0/716901229 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06a006fa40 con 0x7f06a00af900 2024-03-29T10:41:29.163 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:41:29.163 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:29.163 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:29.180+0000 7f0695ffb640 1 -- 172.21.15.57:0/716901229 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f0698052590 con 0x7f06a00af900 2024-03-29T10:41:29.164 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:29.180+0000 7f0695ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:29.164 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:29.180+0000 7f0695ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:29.166 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:29.183+0000 7f06a7833640 10 librados: watch_flush enter 2024-03-29T10:41:29.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:29.184+0000 7f06a7833640 10 librados: watch_flush exit 2024-03-29T10:41:29.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:29.184+0000 7f06a7833640 20 client.4287.objecter shutdown clearing up homeless session... 2024-03-29T10:41:29.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:29.184+0000 7f06a7833640 10 client.4287.objecter successfully canceled tick 2024-03-29T10:41:29.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:29.184+0000 7f06a7833640 1 -- 172.21.15.57:0/716901229 >> v1:172.21.15.179:6816/2698295936 conn(0x7f068406dee0 legacy=0x7f06840703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:29.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:29.184+0000 7f06a7833640 10 monclient: shutdown 2024-03-29T10:41:29.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:29.184+0000 7f06a7833640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:29.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:29.184+0000 7f06a7833640 1 -- 172.21.15.57:0/716901229 >> v1:172.21.15.57:6789/0 conn(0x7f06a00af900 legacy=0x7f06a0071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:29.170 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:29.186+0000 7f06a7833640 1 -- 172.21.15.57:0/716901229 shutdown_connections 2024-03-29T10:41:29.170 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:29.186+0000 7f06a7833640 1 -- 172.21.15.57:0/716901229 >> 172.21.15.57:0/716901229 conn(0x7f06a00b35e0 msgr2=0x7f06a00adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:29.170 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:29.186+0000 7f06a7833640 1 -- 172.21.15.57:0/716901229 shutdown_connections 2024-03-29T10:41:29.170 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:29.186+0000 7f06a7833640 1 -- 172.21.15.57:0/716901229 wait complete. 2024-03-29T10:41:29.172 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:29.189+0000 7f06a7833640 1 librados: shutdown 2024-03-29T10:41:29.189 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:35.996 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:36.148 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.164+0000 7f6606431640 10 monclient: get_monmap_and_config 2024-03-29T10:41:36.148 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.164+0000 7f6606431640 10 monclient: build_initial_monmap 2024-03-29T10:41:36.148 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.164+0000 7f6606431640 10 monclient: monmap: 2024-03-29T10:41:36.148 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:41:36.148 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:36.148 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:41:36.165571+0000 2024-03-29T10:41:36.148 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:41:36.165571+0000 2024-03-29T10:41:36.148 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:36.148 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:36.148 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:41:36.149 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:41:36.149 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:41:36.149 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:36.149 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.166+0000 7f6606431640 1 Processor -- start 2024-03-29T10:41:36.149 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.166+0000 7f6606431640 1 -- start start 2024-03-29T10:41:36.149 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.166+0000 7f6606431640 10 monclient: init 2024-03-29T10:41:36.150 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.167+0000 7f6606431640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:36.150 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.167+0000 7f6606431640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:41:36.154 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.171+0000 7f6606431640 10 monclient(hunting): picked mon.noname-b con 0x7f65f80389e0 addr v1:172.21.15.57:6790/0 2024-03-29T10:41:36.154 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.171+0000 7f6606431640 10 monclient(hunting): picked mon.noname-c con 0x7f65f8038dc0 addr v1:172.21.15.179:6789/0 2024-03-29T10:41:36.154 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.171+0000 7f660542f640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f65f80389e0 0x7f65f8033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:52633/0 (socket says 172.21.15.57:52633) 2024-03-29T10:41:36.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.174+0000 7f6606431640 10 monclient(hunting): picked mon.noname-a con 0x7f65f8030660 addr v1:172.21.15.57:6789/0 2024-03-29T10:41:36.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.174+0000 7f660542f640 1 -- 172.21.15.57:0/2099126523 learned_addr learned my addr 172.21.15.57:0/2099126523 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:41:36.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.175+0000 7f6606431640 1 -- 172.21.15.57:0/2099126523 --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f80c54c0 con 0x7f65f8030660 2024-03-29T10:41:36.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.175+0000 7f6606431640 1 -- 172.21.15.57:0/2099126523 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f8036430 con 0x7f65f8038dc0 2024-03-29T10:41:36.159 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.175+0000 7f6606431640 1 -- 172.21.15.57:0/2099126523 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f8039840 con 0x7f65f80389e0 2024-03-29T10:41:36.159 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.175+0000 7f6606431640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:36.159 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.175+0000 7f6606431640 10 monclient(hunting): authenticate will time out at 1048.351535s 2024-03-29T10:41:36.163 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 1 -- 172.21.15.57:0/2099126523 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290799816 0 0) 0x7f65f80c54c0 con 0x7f65f8030660 2024-03-29T10:41:36.163 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:36.163 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:36.163 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:36.163 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 10 monclient(hunting): my global_id is 4290 2024-03-29T10:41:36.163 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 1 -- 172.21.15.57:0/2099126523 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65ec003520 con 0x7f65f8030660 2024-03-29T10:41:36.163 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 1 -- 172.21.15.57:0/2099126523 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2421488915 0 0) 0x7f65ec003520 con 0x7f65f8030660 2024-03-29T10:41:36.163 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 1 -- 172.21.15.57:0/2099126523 >> v1:172.21.15.57:6790/0 conn(0x7f65f80389e0 legacy=0x7f65f8033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:36.163 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 1 -- 172.21.15.57:0/2099126523 >> v1:172.21.15.179:6789/0 conn(0x7f65f8038dc0 legacy=0x7f65f80cadd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:41:36.164 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:41:36.164 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 1 monclient: found mon.noname-a 2024-03-29T10:41:36.164 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:36.164 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:41:36.164 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 1 -- 172.21.15.57:0/2099126523 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f8035690 con 0x7f65f8030660 2024-03-29T10:41:36.164 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 10 monclient: _finish_auth 0 2024-03-29T10:41:36.165 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:41:36.165 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:36.165 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 1 -- 172.21.15.57:0/2099126523 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f660004f730 con 0x7f65f8030660 2024-03-29T10:41:36.165 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:36.165 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:36.165 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 10 monclient: dump: 2024-03-29T10:41:36.165 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:36.165 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:36.165 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:36.165 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:36.165 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:36.165 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:36.165 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:36.165 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:36.165 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:36.166 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:36.166 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 1 -- 172.21.15.57:0/2099126523 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66000008d0 con 0x7f65f8030660 2024-03-29T10:41:36.166 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:36.166 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 1 -- 172.21.15.57:0/2099126523 <== mon.0 v1:172.21.15.57:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f66000758b0 con 0x7f65f8030660 2024-03-29T10:41:36.166 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:36.166 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:36.166 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.179+0000 7f65f7fff640 10 monclient: dump: 2024-03-29T10:41:36.166 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:36.166 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:36.166 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:36.166 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:36.166 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:36.166 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:36.166 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:36.167 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:36.167 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:36.167 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:36.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.180+0000 7f6606431640 5 monclient: authenticate success, global_id 4290 2024-03-29T10:41:36.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.180+0000 7f6606431640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:36.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.180+0000 7f6606431640 4 set_mon_vals no callback set 2024-03-29T10:41:36.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.181+0000 7f6606431640 10 monclient: shutdown 2024-03-29T10:41:36.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.181+0000 7f6606431640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:36.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.181+0000 7f6606431640 1 -- 172.21.15.57:0/2099126523 >> v1:172.21.15.57:6789/0 conn(0x7f65f8030660 legacy=0x7f65f80d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:36.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.181+0000 7f6606431640 1 -- 172.21.15.57:0/2099126523 shutdown_connections 2024-03-29T10:41:36.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.181+0000 7f6606431640 1 -- 172.21.15.57:0/2099126523 >> 172.21.15.57:0/2099126523 conn(0x7f65f8005c00 msgr2=0x7f65f8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:36.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.181+0000 7f6606431640 1 -- 172.21.15.57:0/2099126523 shutdown_connections 2024-03-29T10:41:36.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.181+0000 7f6606431640 1 -- 172.21.15.57:0/2099126523 wait complete. 2024-03-29T10:41:36.168 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.182+0000 7f6606431640 10 monclient: build_initial_monmap 2024-03-29T10:41:36.168 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.182+0000 7f6606431640 10 monclient: monmap: 2024-03-29T10:41:36.168 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:41:36.169 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:36.169 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:41:36.169 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:41:36.169 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:36.169 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:36.169 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:41:36.169 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:41:36.169 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:41:36.169 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:36.169 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.182+0000 7f6606431640 1 librados: starting msgr at 2024-03-29T10:41:36.169 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.182+0000 7f6606431640 1 librados: starting objecter 2024-03-29T10:41:36.169 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.182+0000 7f6606431640 1 Processor -- start 2024-03-29T10:41:36.169 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.183+0000 7f6606431640 1 -- start start 2024-03-29T10:41:36.170 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.183+0000 7f6606431640 1 librados: setting wanted keys 2024-03-29T10:41:36.170 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.183+0000 7f6606431640 1 librados: calling monclient init 2024-03-29T10:41:36.170 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.183+0000 7f6606431640 10 monclient: init 2024-03-29T10:41:36.170 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.183+0000 7f6606431640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:36.170 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.183+0000 7f6606431640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:36.171 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.186+0000 7f6606431640 10 monclient(hunting): picked mon.noname-b con 0x7f65f8030660 addr v1:172.21.15.179:6789/0 2024-03-29T10:41:36.171 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.187+0000 7f6606431640 10 monclient(hunting): picked mon.noname-a con 0x7f65f80389e0 addr v1:172.21.15.57:6789/0 2024-03-29T10:41:36.171 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.187+0000 7f6606431640 10 monclient(hunting): picked mon.noname-c con 0x7f65f80c7ea0 addr v1:172.21.15.57:6790/0 2024-03-29T10:41:36.171 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.187+0000 7f6606431640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f81646b0 con 0x7f65f80389e0 2024-03-29T10:41:36.171 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.187+0000 7f6606431640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f80c0b90 con 0x7f65f8030660 2024-03-29T10:41:36.171 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.187+0000 7f6606431640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f8164310 con 0x7f65f80c7ea0 2024-03-29T10:41:36.171 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.187+0000 7f6606431640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:36.171 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.187+0000 7f6606431640 10 monclient(hunting): authenticate will time out at 1048.363301s 2024-03-29T10:41:36.174 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.190+0000 7f6604c2e640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f65f80389e0 0x7f65f8161680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:33563/0 (socket says 172.21.15.57:33563) 2024-03-29T10:41:36.174 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.190+0000 7f6604c2e640 1 -- 172.21.15.57:0/345917354 learned_addr learned my addr 172.21.15.57:0/345917354 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:41:36.174 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.190+0000 7f65f5ffb640 10 client.?.objecter ms_handle_connect 0x7f65f80389e0 2024-03-29T10:41:36.174 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.190+0000 7f65f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:36.174 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.191+0000 7f65f5ffb640 1 -- 172.21.15.57:0/345917354 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342883706 0 0) 0x7f65f81646b0 con 0x7f65f80389e0 2024-03-29T10:41:36.174 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.191+0000 7f65f5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:41:36.174 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.191+0000 7f65f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:36.174 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.191+0000 7f65f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:36.174 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.191+0000 7f65f5ffb640 10 monclient(hunting): my global_id is 4293 2024-03-29T10:41:36.174 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.191+0000 7f65f5ffb640 1 -- 172.21.15.57:0/345917354 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65e4004eb0 con 0x7f65f80389e0 2024-03-29T10:41:36.175 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.191+0000 7f65f5ffb640 10 client.?.objecter ms_handle_connect 0x7f65f8030660 2024-03-29T10:41:36.175 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.191+0000 7f65f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:36.175 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.191+0000 7f65f5ffb640 1 -- 172.21.15.57:0/345917354 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1919506080 0 0) 0x7f65e4004eb0 con 0x7f65f80389e0 2024-03-29T10:41:36.175 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.191+0000 7f65f5ffb640 1 -- 172.21.15.57:0/345917354 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65f81646b0 con 0x7f65f80389e0 2024-03-29T10:41:36.175 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.191+0000 7f65f5ffb640 1 -- 172.21.15.57:0/345917354 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f65f00017d0 con 0x7f65f80389e0 2024-03-29T10:41:36.175 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.191+0000 7f65f5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:36.175 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.191+0000 7f65f5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:36.175 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.191+0000 7f65f5ffb640 10 monclient(hunting): dump: 2024-03-29T10:41:36.176 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:36.176 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:36.176 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:36.176 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:36.176 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:36.176 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:36.176 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:36.176 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:36.176 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:36.176 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:36.176 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.191+0000 7f65f5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:36.178 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.194+0000 7f65f5ffb640 10 client.?.objecter ms_handle_connect 0x7f65f80c7ea0 2024-03-29T10:41:36.179 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.194+0000 7f65f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:36.179 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.194+0000 7f65f5ffb640 1 -- 172.21.15.57:0/345917354 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569651749 0 0) 0x7f65f80c0b90 con 0x7f65f8030660 2024-03-29T10:41:36.179 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.194+0000 7f65f5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:41:36.179 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.194+0000 7f65f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:36.179 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.194+0000 7f65f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:36.179 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.194+0000 7f65f5ffb640 10 monclient(hunting): my global_id is 4300 2024-03-29T10:41:36.179 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.194+0000 7f65f5ffb640 1 -- 172.21.15.57:0/345917354 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65e4004eb0 con 0x7f65f8030660 2024-03-29T10:41:36.179 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.194+0000 7f65f5ffb640 1 -- 172.21.15.57:0/345917354 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2153128061 0 0) 0x7f65f81646b0 con 0x7f65f80389e0 2024-03-29T10:41:36.179 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.194+0000 7f65f5ffb640 1 -- 172.21.15.57:0/345917354 >> v1:172.21.15.57:6790/0 conn(0x7f65f80c7ea0 legacy=0x7f65f802d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:36.179 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.194+0000 7f65f5ffb640 1 -- 172.21.15.57:0/345917354 >> v1:172.21.15.179:6789/0 conn(0x7f65f8030660 legacy=0x7f65f8164bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:36.179 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.194+0000 7f65f5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:41:36.179 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.194+0000 7f65f5ffb640 1 monclient: found mon.a 2024-03-29T10:41:36.179 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.194+0000 7f65f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:36.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.194+0000 7f65f5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:41:36.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.194+0000 7f65f5ffb640 1 -- 172.21.15.57:0/345917354 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f802d860 con 0x7f65f80389e0 2024-03-29T10:41:36.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.195+0000 7f65f5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:41:36.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.195+0000 7f65f5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:41:36.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.195+0000 7f65f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:36.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.195+0000 7f6606431640 5 monclient: authenticate success, global_id 4293 2024-03-29T10:41:36.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.195+0000 7f6606431640 10 monclient: _renew_subs 2024-03-29T10:41:36.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.195+0000 7f6606431640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:41:36.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.195+0000 7f6606431640 1 -- 172.21.15.57:0/345917354 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f802dab0 con 0x7f65f80389e0 2024-03-29T10:41:36.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.195+0000 7f65f5ffb640 1 -- 172.21.15.57:0/345917354 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65f0001930 con 0x7f65f80389e0 2024-03-29T10:41:36.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.195+0000 7f6606431640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:36.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.195+0000 7f6606431640 10 monclient: _renew_subs 2024-03-29T10:41:36.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.195+0000 7f6606431640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:41:36.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.195+0000 7f6606431640 1 -- 172.21.15.57:0/345917354 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f8160ff0 con 0x7f65f80389e0 2024-03-29T10:41:36.181 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.195+0000 7f6606431640 1 librados: init done 2024-03-29T10:41:36.182 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.196+0000 7f65f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:36.182 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.196+0000 7f65f77fe640 4 set_mon_vals no callback set 2024-03-29T10:41:36.182 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.197+0000 7f65f5ffb640 1 -- 172.21.15.57:0/345917354 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f65f0004a00 con 0x7f65f80389e0 2024-03-29T10:41:36.182 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.197+0000 7f65f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:36.182 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.197+0000 7f65f5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:36.182 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.197+0000 7f65f5ffb640 10 monclient: dump: 2024-03-29T10:41:36.182 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:36.182 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:36.183 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:36.183 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:36.183 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:36.183 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:36.183 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:36.183 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:36.183 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:36.183 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:36.183 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.198+0000 7f65f5ffb640 1 -- 172.21.15.57:0/345917354 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f65f001aa10 con 0x7f65f80389e0 2024-03-29T10:41:36.183 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.198+0000 7f65f5ffb640 1 -- 172.21.15.57:0/345917354 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f65f0084a40 con 0x7f65f80389e0 2024-03-29T10:41:36.183 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.198+0000 7f65f5ffb640 10 client.4293.objecter ms_dispatch 0x7f6600000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:36.183 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.198+0000 7f65f5ffb640 3 client.4293.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:36.183 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.198+0000 7f65f5ffb640 3 client.4293.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:36.183 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.198+0000 7f65f5ffb640 20 client.4293.objecter dump_active .. 0 homeless 2024-03-29T10:41:36.183 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.198+0000 7f6606431640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:36.184 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.198+0000 7f6606431640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:36.184 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.198+0000 7f6606431640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:41:36.184 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.198+0000 7f6606431640 1 -- 172.21.15.57:0/345917354 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e0005080 con 0x7f65f80389e0 2024-03-29T10:41:36.184 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.199+0000 7f660542f640 10 monclient: get_auth_request con 0x7f65e406dc00 auth_method 0 2024-03-29T10:41:36.184 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.199+0000 7f65f5ffb640 10 client.4293.objecter ms_handle_connect 0x7f65e406dc00 2024-03-29T10:41:36.190 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.207+0000 7f65f5ffb640 1 -- 172.21.15.57:0/345917354 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f65f00503a0 con 0x7f65f80389e0 2024-03-29T10:41:36.190 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.207+0000 7f65f5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:36.190 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.207+0000 7f65f5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:36.521 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.537+0000 7f6606431640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:36.521 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.537+0000 7f6606431640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:36.521 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.538+0000 7f6606431640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:41:36.521 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.538+0000 7f6606431640 1 -- 172.21.15.57:0/345917354 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65e0005330 con 0x7f65f80389e0 2024-03-29T10:41:36.523 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.540+0000 7f65f5ffb640 1 -- 172.21.15.57:0/345917354 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f65f00522d0 con 0x7f65f80389e0 2024-03-29T10:41:36.524 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.540+0000 7f65f5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:36.524 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.540+0000 7f65f5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:36.524 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:41:36.525 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:36.530 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.545+0000 7f6606431640 10 librados: watch_flush enter 2024-03-29T10:41:36.530 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.545+0000 7f6606431640 10 librados: watch_flush exit 2024-03-29T10:41:36.530 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.545+0000 7f6606431640 20 client.4293.objecter shutdown clearing up homeless session... 2024-03-29T10:41:36.530 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.545+0000 7f6606431640 10 client.4293.objecter successfully canceled tick 2024-03-29T10:41:36.530 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.545+0000 7f6606431640 1 -- 172.21.15.57:0/345917354 >> v1:172.21.15.179:6816/2698295936 conn(0x7f65e406dc00 legacy=0x7f65e40700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:36.530 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.545+0000 7f6606431640 10 monclient: shutdown 2024-03-29T10:41:36.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.545+0000 7f6606431640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:36.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.545+0000 7f6606431640 1 -- 172.21.15.57:0/345917354 >> v1:172.21.15.57:6789/0 conn(0x7f65f80389e0 legacy=0x7f65f8161680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:36.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.547+0000 7f6606431640 1 -- 172.21.15.57:0/345917354 shutdown_connections 2024-03-29T10:41:36.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.547+0000 7f6606431640 1 -- 172.21.15.57:0/345917354 >> 172.21.15.57:0/345917354 conn(0x7f65f8005c00 msgr2=0x7f65f8034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:36.533 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.550+0000 7f6606431640 1 -- 172.21.15.57:0/345917354 shutdown_connections 2024-03-29T10:41:36.533 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.550+0000 7f6606431640 1 -- 172.21.15.57:0/345917354 wait complete. 2024-03-29T10:41:36.533 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:36.550+0000 7f6606431640 1 librados: shutdown 2024-03-29T10:41:36.550 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:43.357 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:43.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.523+0000 7fd1415b9640 10 monclient: get_monmap_and_config 2024-03-29T10:41:43.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.523+0000 7fd1415b9640 10 monclient: build_initial_monmap 2024-03-29T10:41:43.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.523+0000 7fd1415b9640 10 monclient: monmap: 2024-03-29T10:41:43.508 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:41:43.509 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:43.509 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:41:43.524072+0000 2024-03-29T10:41:43.509 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:41:43.524072+0000 2024-03-29T10:41:43.509 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:43.509 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:43.509 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:41:43.509 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:41:43.509 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:41:43.509 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:43.509 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.524+0000 7fd1415b9640 1 Processor -- start 2024-03-29T10:41:43.509 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.524+0000 7fd1415b9640 1 -- start start 2024-03-29T10:41:43.509 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.524+0000 7fd1415b9640 10 monclient: init 2024-03-29T10:41:43.509 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.525+0000 7fd1415b9640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:43.509 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.525+0000 7fd1415b9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:41:43.509 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.525+0000 7fd1415b9640 10 monclient(hunting): picked mon.noname-a con 0x7fd13c0af6c0 addr v1:172.21.15.57:6789/0 2024-03-29T10:41:43.510 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.525+0000 7fd1415b9640 10 monclient(hunting): picked mon.noname-b con 0x7fd13c14f6b0 addr v1:172.21.15.57:6790/0 2024-03-29T10:41:43.510 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.525+0000 7fd1415b9640 10 monclient(hunting): picked mon.noname-c con 0x7fd13c152980 addr v1:172.21.15.179:6789/0 2024-03-29T10:41:43.510 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.525+0000 7fd1415b9640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd13c0a94e0 con 0x7fd13c0af6c0 2024-03-29T10:41:43.510 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.525+0000 7fd1415b9640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd13c0afb60 con 0x7fd13c152980 2024-03-29T10:41:43.510 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.525+0000 7fd1415b9640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd13c0aab80 con 0x7fd13c14f6b0 2024-03-29T10:41:43.510 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.525+0000 7fd1415b9640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:43.510 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.525+0000 7fd1415b9640 10 monclient(hunting): authenticate will time out at 1055.701666s 2024-03-29T10:41:43.512 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.529+0000 7fd13b7fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fd13c152980 0x7fd13c152e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:54601/0 (socket says 172.21.15.57:54601) 2024-03-29T10:41:43.512 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.529+0000 7fd13b7fe640 1 -- 172.21.15.57:0/866502751 learned_addr learned my addr 172.21.15.57:0/866502751 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:41:43.515 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.530+0000 7fd139ffb640 1 -- 172.21.15.57:0/866502751 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 899331501 0 0) 0x7fd13c0afb60 con 0x7fd13c152980 2024-03-29T10:41:43.515 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.530+0000 7fd139ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:41:43.515 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.530+0000 7fd139ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:43.516 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.530+0000 7fd139ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:43.516 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.530+0000 7fd139ffb640 10 monclient(hunting): my global_id is 4303 2024-03-29T10:41:43.516 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.530+0000 7fd139ffb640 1 -- 172.21.15.57:0/866502751 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd124003520 con 0x7fd13c152980 2024-03-29T10:41:43.516 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.530+0000 7fd139ffb640 1 -- 172.21.15.57:0/866502751 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 336427270 0 0) 0x7fd13c0a94e0 con 0x7fd13c0af6c0 2024-03-29T10:41:43.516 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.530+0000 7fd139ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:41:43.516 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.530+0000 7fd139ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:43.516 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.530+0000 7fd139ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:43.516 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.530+0000 7fd139ffb640 10 monclient(hunting): my global_id is 4296 2024-03-29T10:41:43.516 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.530+0000 7fd139ffb640 1 -- 172.21.15.57:0/866502751 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd124003840 con 0x7fd13c0af6c0 2024-03-29T10:41:43.516 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.531+0000 7fd139ffb640 1 -- 172.21.15.57:0/866502751 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2373919435 0 0) 0x7fd124003520 con 0x7fd13c152980 2024-03-29T10:41:43.516 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.531+0000 7fd139ffb640 1 -- 172.21.15.57:0/866502751 >> v1:172.21.15.57:6790/0 conn(0x7fd13c14f6b0 legacy=0x7fd13c14fae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:43.516 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.531+0000 7fd139ffb640 1 -- 172.21.15.57:0/866502751 >> v1:172.21.15.57:6789/0 conn(0x7fd13c0af6c0 legacy=0x7fd13c0ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:43.516 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.531+0000 7fd139ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:41:43.516 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.531+0000 7fd139ffb640 1 monclient: found mon.noname-c 2024-03-29T10:41:43.517 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.531+0000 7fd139ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:43.517 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.531+0000 7fd139ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:41:43.517 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.531+0000 7fd139ffb640 1 -- 172.21.15.57:0/866502751 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13c0aa680 con 0x7fd13c152980 2024-03-29T10:41:43.517 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.531+0000 7fd139ffb640 10 monclient: _finish_auth 0 2024-03-29T10:41:43.517 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.531+0000 7fd139ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:41:43.517 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.531+0000 7fd139ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:43.517 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.531+0000 7fd139ffb640 1 -- 172.21.15.57:0/866502751 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd130001670 con 0x7fd13c152980 2024-03-29T10:41:43.517 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.531+0000 7fd139ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:43.517 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.531+0000 7fd139ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:43.517 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.531+0000 7fd139ffb640 10 monclient: dump: 2024-03-29T10:41:43.517 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:43.517 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:43.517 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:43.517 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:43.518 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:43.518 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:43.518 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:43.518 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:43.518 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:43.518 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:43.518 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.532+0000 7fd1415b9640 5 monclient: authenticate success, global_id 4303 2024-03-29T10:41:43.518 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.532+0000 7fd1415b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:43.519 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.533+0000 7fd139ffb640 1 -- 172.21.15.57:0/866502751 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd130001850 con 0x7fd13c152980 2024-03-29T10:41:43.519 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.533+0000 7fd139ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:43.519 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.533+0000 7fd139ffb640 1 -- 172.21.15.57:0/866502751 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd130001c60 con 0x7fd13c152980 2024-03-29T10:41:43.519 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.533+0000 7fd139ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:43.519 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.533+0000 7fd139ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:43.519 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.533+0000 7fd139ffb640 10 monclient: dump: 2024-03-29T10:41:43.519 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:43.519 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:43.519 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:43.519 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:43.519 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:43.520 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:43.520 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:43.520 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:43.520 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:43.520 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:43.520 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.533+0000 7fd1415b9640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:43.520 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.533+0000 7fd1415b9640 4 set_mon_vals no callback set 2024-03-29T10:41:43.520 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.536+0000 7fd1415b9640 10 monclient: shutdown 2024-03-29T10:41:43.521 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.536+0000 7fd1415b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:43.521 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.536+0000 7fd1415b9640 1 -- 172.21.15.57:0/866502751 >> v1:172.21.15.179:6789/0 conn(0x7fd13c152980 legacy=0x7fd13c152e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:43.522 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.539+0000 7fd1415b9640 1 -- 172.21.15.57:0/866502751 shutdown_connections 2024-03-29T10:41:43.522 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.539+0000 7fd1415b9640 1 -- 172.21.15.57:0/866502751 >> 172.21.15.57:0/866502751 conn(0x7fd13c0b35e0 msgr2=0x7fd13c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:43.523 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.540+0000 7fd1415b9640 1 -- 172.21.15.57:0/866502751 shutdown_connections 2024-03-29T10:41:43.523 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.540+0000 7fd1415b9640 1 -- 172.21.15.57:0/866502751 wait complete. 2024-03-29T10:41:43.527 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.541+0000 7fd1415b9640 10 monclient: build_initial_monmap 2024-03-29T10:41:43.527 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.541+0000 7fd1415b9640 10 monclient: monmap: 2024-03-29T10:41:43.527 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:41:43.527 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:43.527 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:41:43.528 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:41:43.528 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:43.528 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:43.528 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:41:43.528 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:41:43.528 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:41:43.528 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:43.528 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.542+0000 7fd1415b9640 1 librados: starting msgr at 2024-03-29T10:41:43.528 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.542+0000 7fd1415b9640 1 librados: starting objecter 2024-03-29T10:41:43.528 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.542+0000 7fd1415b9640 1 Processor -- start 2024-03-29T10:41:43.528 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.542+0000 7fd1415b9640 1 -- start start 2024-03-29T10:41:43.528 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.542+0000 7fd1415b9640 1 librados: setting wanted keys 2024-03-29T10:41:43.528 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.542+0000 7fd1415b9640 1 librados: calling monclient init 2024-03-29T10:41:43.528 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.542+0000 7fd1415b9640 10 monclient: init 2024-03-29T10:41:43.528 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.543+0000 7fd1415b9640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:43.529 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.543+0000 7fd1415b9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:41:43.529 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.543+0000 7fd1415b9640 10 monclient(hunting): picked mon.noname-c con 0x7fd13c0af6c0 addr v1:172.21.15.57:6790/0 2024-03-29T10:41:43.529 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.543+0000 7fd1415b9640 10 monclient(hunting): picked mon.noname-a con 0x7fd13c14f6b0 addr v1:172.21.15.57:6789/0 2024-03-29T10:41:43.529 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.543+0000 7fd1415b9640 10 monclient(hunting): picked mon.noname-b con 0x7fd13c152980 addr v1:172.21.15.179:6789/0 2024-03-29T10:41:43.529 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.543+0000 7fd1415b9640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd13c086580 con 0x7fd13c14f6b0 2024-03-29T10:41:43.529 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.543+0000 7fd1415b9640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd13c14f100 con 0x7fd13c152980 2024-03-29T10:41:43.529 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.543+0000 7fd1415b9640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd13c087890 con 0x7fd13c0af6c0 2024-03-29T10:41:43.529 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.543+0000 7fd1415b9640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:43.529 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.543+0000 7fd1415b9640 10 monclient(hunting): authenticate will time out at 1055.719611s 2024-03-29T10:41:43.530 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.547+0000 7fd13b7fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fd13c152980 0x7fd13c14e9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:54703/0 (socket says 172.21.15.57:54703) 2024-03-29T10:41:43.530 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.547+0000 7fd13b7fe640 1 -- 172.21.15.57:0/1709793053 learned_addr learned my addr 172.21.15.57:0/1709793053 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:41:43.530 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.547+0000 7fd1237fe640 10 client.?.objecter ms_handle_connect 0x7fd13c152980 2024-03-29T10:41:43.530 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.547+0000 7fd1237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:43.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.547+0000 7fd1237fe640 1 -- 172.21.15.57:0/1709793053 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3534885370 0 0) 0x7fd13c14f100 con 0x7fd13c152980 2024-03-29T10:41:43.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.547+0000 7fd1237fe640 10 monclient(hunting): _negotiate 2024-03-29T10:41:43.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.547+0000 7fd1237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:43.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.547+0000 7fd1237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:43.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.547+0000 7fd1237fe640 10 monclient(hunting): my global_id is 4306 2024-03-29T10:41:43.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.547+0000 7fd1237fe640 1 -- 172.21.15.57:0/1709793053 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd128004eb0 con 0x7fd13c152980 2024-03-29T10:41:43.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.548+0000 7fd1237fe640 1 -- 172.21.15.57:0/1709793053 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2711430996 0 0) 0x7fd128004eb0 con 0x7fd13c152980 2024-03-29T10:41:43.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.548+0000 7fd1237fe640 1 -- 172.21.15.57:0/1709793053 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd13c14f100 con 0x7fd13c152980 2024-03-29T10:41:43.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.548+0000 7fd1237fe640 1 -- 172.21.15.57:0/1709793053 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd130001c60 con 0x7fd13c152980 2024-03-29T10:41:43.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.548+0000 7fd1237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:43.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.548+0000 7fd1237fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:43.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.548+0000 7fd1237fe640 10 monclient(hunting): dump: 2024-03-29T10:41:43.531 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:43.532 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:43.532 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:43.532 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:43.532 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:43.532 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:43.532 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:43.532 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:43.532 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:43.532 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:43.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.548+0000 7fd1237fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:43.533 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.549+0000 7fd1237fe640 1 -- 172.21.15.57:0/1709793053 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3940246491 0 0) 0x7fd13c14f100 con 0x7fd13c152980 2024-03-29T10:41:43.533 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.549+0000 7fd1237fe640 1 -- 172.21.15.57:0/1709793053 >> v1:172.21.15.57:6790/0 conn(0x7fd13c0af6c0 legacy=0x7fd13c1efc60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:41:43.534 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.549+0000 7fd1237fe640 1 -- 172.21.15.57:0/1709793053 >> v1:172.21.15.57:6789/0 conn(0x7fd13c14f6b0 legacy=0x7fd13c14e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:43.534 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.549+0000 7fd1237fe640 10 monclient: _finish_hunting 0 2024-03-29T10:41:43.534 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.549+0000 7fd1237fe640 1 monclient: found mon.b 2024-03-29T10:41:43.534 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.549+0000 7fd1237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:43.534 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.549+0000 7fd1237fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:43.534 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.549+0000 7fd1237fe640 1 -- 172.21.15.57:0/1709793053 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13c085510 con 0x7fd13c152980 2024-03-29T10:41:43.534 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.549+0000 7fd1237fe640 10 monclient: _finish_auth 0 2024-03-29T10:41:43.534 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.549+0000 7fd1237fe640 10 monclient: _check_auth_tickets 2024-03-29T10:41:43.534 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.549+0000 7fd1237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:43.534 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.549+0000 7fd1237fe640 10 client.?.objecter ms_handle_connect 0x7fd13c14f6b0 2024-03-29T10:41:43.534 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.549+0000 7fd1237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:43.536 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.551+0000 7fd1415b9640 5 monclient: authenticate success, global_id 4306 2024-03-29T10:41:43.536 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.551+0000 7fd1415b9640 10 monclient: _renew_subs 2024-03-29T10:41:43.536 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.551+0000 7fd1415b9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:43.536 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.551+0000 7fd1415b9640 1 -- 172.21.15.57:0/1709793053 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd13c0857c0 con 0x7fd13c152980 2024-03-29T10:41:43.536 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.551+0000 7fd1415b9640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:43.536 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.551+0000 7fd1415b9640 10 monclient: _renew_subs 2024-03-29T10:41:43.536 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.551+0000 7fd1415b9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:43.536 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.551+0000 7fd1415b9640 1 -- 172.21.15.57:0/1709793053 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd13c086120 con 0x7fd13c152980 2024-03-29T10:41:43.536 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.551+0000 7fd1415b9640 1 librados: init done 2024-03-29T10:41:43.537 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.552+0000 7fd1237fe640 1 -- 172.21.15.57:0/1709793053 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd130001180 con 0x7fd13c152980 2024-03-29T10:41:43.537 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.552+0000 7fd1237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:43.537 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.552+0000 7fd1237fe640 1 -- 172.21.15.57:0/1709793053 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd130004c70 con 0x7fd13c152980 2024-03-29T10:41:43.537 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.552+0000 7fd1237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:43.537 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.552+0000 7fd1237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:43.537 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.552+0000 7fd1237fe640 10 monclient: dump: 2024-03-29T10:41:43.537 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:43.537 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:43.537 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:43.537 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:43.537 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:43.537 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:43.537 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:43.537 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:43.537 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:43.538 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:43.538 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.552+0000 7fd1397fa640 4 set_mon_vals no callback set 2024-03-29T10:41:43.539 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.554+0000 7fd1415b9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:43.539 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.555+0000 7fd1415b9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:43.539 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.555+0000 7fd1415b9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:43.539 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.555+0000 7fd1415b9640 1 -- 172.21.15.57:0/1709793053 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd11c005080 con 0x7fd13c152980 2024-03-29T10:41:43.539 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.555+0000 7fd1237fe640 1 -- 172.21.15.57:0/1709793053 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fd13001ac40 con 0x7fd13c152980 2024-03-29T10:41:43.539 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.556+0000 7fd1237fe640 1 -- 172.21.15.57:0/1709793053 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7fd130084b80 con 0x7fd13c152980 2024-03-29T10:41:43.539 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.556+0000 7fd1237fe640 10 client.4306.objecter ms_dispatch 0x7fd13c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:43.539 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.556+0000 7fd1237fe640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:43.539 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.556+0000 7fd1237fe640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:43.540 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.556+0000 7fd1237fe640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-29T10:41:43.540 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.556+0000 7fd13affd640 10 monclient: get_auth_request con 0x7fd12806db80 auth_method 0 2024-03-29T10:41:43.540 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.557+0000 7fd1237fe640 10 client.4306.objecter ms_handle_connect 0x7fd12806db80 2024-03-29T10:41:43.548 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.565+0000 7fd1237fe640 1 -- 172.21.15.57:0/1709793053 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd130050290 con 0x7fd13c152980 2024-03-29T10:41:43.548 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.565+0000 7fd1237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:43.548 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.565+0000 7fd1237fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:43.891 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.906+0000 7fd1415b9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:43.891 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.906+0000 7fd1415b9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:43.891 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.906+0000 7fd1415b9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:43.891 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.906+0000 7fd1415b9640 1 -- 172.21.15.57:0/1709793053 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd11c005330 con 0x7fd13c152980 2024-03-29T10:41:43.891 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.907+0000 7fd1237fe640 1 -- 172.21.15.57:0/1709793053 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7fd130052390 con 0x7fd13c152980 2024-03-29T10:41:43.891 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.908+0000 7fd1237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:43.891 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.908+0000 7fd1237fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:43.892 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:41:43.892 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:43.895 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.912+0000 7fd1415b9640 10 librados: watch_flush enter 2024-03-29T10:41:43.895 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.912+0000 7fd1415b9640 10 librados: watch_flush exit 2024-03-29T10:41:43.895 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.912+0000 7fd1415b9640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-29T10:41:43.895 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.912+0000 7fd1415b9640 10 client.4306.objecter successfully canceled tick 2024-03-29T10:41:43.897 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.914+0000 7fd1415b9640 1 -- 172.21.15.57:0/1709793053 >> v1:172.21.15.179:6816/2698295936 conn(0x7fd12806db80 legacy=0x7fd128070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:43.898 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.914+0000 7fd1415b9640 10 monclient: shutdown 2024-03-29T10:41:43.898 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.914+0000 7fd1415b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:43.898 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.914+0000 7fd1415b9640 1 -- 172.21.15.57:0/1709793053 >> v1:172.21.15.179:6789/0 conn(0x7fd13c152980 legacy=0x7fd13c14e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:43.899 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.916+0000 7fd1415b9640 1 -- 172.21.15.57:0/1709793053 shutdown_connections 2024-03-29T10:41:43.899 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.916+0000 7fd1415b9640 1 -- 172.21.15.57:0/1709793053 >> 172.21.15.57:0/1709793053 conn(0x7fd13c0b35e0 msgr2=0x7fd13c151af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:43.900 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.916+0000 7fd1415b9640 1 -- 172.21.15.57:0/1709793053 shutdown_connections 2024-03-29T10:41:43.900 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.917+0000 7fd1415b9640 1 -- 172.21.15.57:0/1709793053 wait complete. 2024-03-29T10:41:43.900 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:43.917+0000 7fd1415b9640 1 librados: shutdown 2024-03-29T10:41:43.921 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:50.729 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:50.899 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.916+0000 7fdfc6257640 10 monclient: get_monmap_and_config 2024-03-29T10:41:50.899 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.916+0000 7fdfc6257640 10 monclient: build_initial_monmap 2024-03-29T10:41:50.899 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.916+0000 7fdfc6257640 10 monclient: monmap: 2024-03-29T10:41:50.899 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:41:50.899 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:50.899 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:41:50.916743+0000 2024-03-29T10:41:50.900 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:41:50.916743+0000 2024-03-29T10:41:50.900 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:50.900 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:50.900 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:41:50.900 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:41:50.900 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:41:50.900 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:50.900 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.917+0000 7fdfc6257640 1 Processor -- start 2024-03-29T10:41:50.901 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.917+0000 7fdfc6257640 1 -- start start 2024-03-29T10:41:50.901 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.917+0000 7fdfc6257640 10 monclient: init 2024-03-29T10:41:50.901 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.918+0000 7fdfc6257640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:50.901 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.918+0000 7fdfc6257640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:41:50.902 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.918+0000 7fdfc6257640 10 monclient(hunting): picked mon.noname-a con 0x7fdfc01457f0 addr v1:172.21.15.57:6789/0 2024-03-29T10:41:50.902 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.918+0000 7fdfc6257640 10 monclient(hunting): picked mon.noname-b con 0x7fdfc013f650 addr v1:172.21.15.57:6790/0 2024-03-29T10:41:50.902 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.919+0000 7fdfc6257640 10 monclient(hunting): picked mon.noname-c con 0x7fdfc00a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T10:41:50.902 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.919+0000 7fdfc6257640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc00aaef0 con 0x7fdfc01457f0 2024-03-29T10:41:50.902 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.919+0000 7fdfc6257640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc0145bd0 con 0x7fdfc00a9d70 2024-03-29T10:41:50.902 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.919+0000 7fdfc6257640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc0145eb0 con 0x7fdfc013f650 2024-03-29T10:41:50.902 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.919+0000 7fdfc6257640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:50.902 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.919+0000 7fdfc6257640 10 monclient(hunting): authenticate will time out at 1063.095082s 2024-03-29T10:41:50.903 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.919+0000 7fdfbf7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fdfc01457f0 0x7fdfc013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:33556/0 (socket says 172.21.15.57:33556) 2024-03-29T10:41:50.903 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.919+0000 7fdfbf7fe640 1 -- 172.21.15.57:0/2343472748 learned_addr learned my addr 172.21.15.57:0/2343472748 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:41:50.906 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.920+0000 7fdfbe7fc640 1 -- 172.21.15.57:0/2343472748 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673196048 0 0) 0x7fdfc0145bd0 con 0x7fdfc00a9d70 2024-03-29T10:41:50.906 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.920+0000 7fdfbe7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:50.906 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.920+0000 7fdfbe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:50.906 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.920+0000 7fdfbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:50.907 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.920+0000 7fdfbe7fc640 10 monclient(hunting): my global_id is 4309 2024-03-29T10:41:50.907 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.920+0000 7fdfbe7fc640 1 -- 172.21.15.57:0/2343472748 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfa8003520 con 0x7fdfc00a9d70 2024-03-29T10:41:50.907 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.920+0000 7fdfbe7fc640 1 -- 172.21.15.57:0/2343472748 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251972701 0 0) 0x7fdfc0145eb0 con 0x7fdfc013f650 2024-03-29T10:41:50.907 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.920+0000 7fdfbe7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:50.907 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.920+0000 7fdfbe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:50.907 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.920+0000 7fdfbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:50.907 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.920+0000 7fdfbe7fc640 10 monclient(hunting): my global_id is 4313 2024-03-29T10:41:50.907 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.920+0000 7fdfbe7fc640 1 -- 172.21.15.57:0/2343472748 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfc0145bd0 con 0x7fdfc013f650 2024-03-29T10:41:50.907 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.920+0000 7fdfbe7fc640 1 -- 172.21.15.57:0/2343472748 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217944947 0 0) 0x7fdfc00aaef0 con 0x7fdfc01457f0 2024-03-29T10:41:50.907 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.920+0000 7fdfbe7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:41:50.907 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.920+0000 7fdfbe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:50.907 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.920+0000 7fdfbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:50.907 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.920+0000 7fdfbe7fc640 10 monclient(hunting): my global_id is 4302 2024-03-29T10:41:50.907 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.920+0000 7fdfbe7fc640 1 -- 172.21.15.57:0/2343472748 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfc0145eb0 con 0x7fdfc01457f0 2024-03-29T10:41:50.908 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.921+0000 7fdfbe7fc640 1 -- 172.21.15.57:0/2343472748 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 333000028 0 0) 0x7fdfa8003520 con 0x7fdfc00a9d70 2024-03-29T10:41:50.908 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.921+0000 7fdfbe7fc640 1 -- 172.21.15.57:0/2343472748 >> v1:172.21.15.57:6790/0 conn(0x7fdfc013f650 legacy=0x7fdfc00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:50.908 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.921+0000 7fdfbe7fc640 1 -- 172.21.15.57:0/2343472748 >> v1:172.21.15.57:6789/0 conn(0x7fdfc01457f0 legacy=0x7fdfc013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:50.908 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.921+0000 7fdfbe7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:41:50.908 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.921+0000 7fdfbe7fc640 1 monclient: found mon.noname-c 2024-03-29T10:41:50.908 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.921+0000 7fdfbe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:50.908 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.921+0000 7fdfbe7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:41:50.908 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.921+0000 7fdfbe7fc640 1 -- 172.21.15.57:0/2343472748 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc0146340 con 0x7fdfc00a9d70 2024-03-29T10:41:50.908 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.921+0000 7fdfbe7fc640 10 monclient: _finish_auth 0 2024-03-29T10:41:50.908 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.921+0000 7fdfbe7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:41:50.908 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.921+0000 7fdfbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:50.908 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.921+0000 7fdfbe7fc640 1 -- 172.21.15.57:0/2343472748 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fdfb00015c0 con 0x7fdfc00a9d70 2024-03-29T10:41:50.908 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.921+0000 7fdfc6257640 5 monclient: authenticate success, global_id 4309 2024-03-29T10:41:50.908 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.921+0000 7fdfc6257640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:50.909 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.921+0000 7fdfbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:50.909 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.921+0000 7fdfbe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:50.909 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.921+0000 7fdfbe7fc640 10 monclient: dump: 2024-03-29T10:41:50.909 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:50.909 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:50.909 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:50.909 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:50.909 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:50.909 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:50.909 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:50.909 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:50.909 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:50.909 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:50.909 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.921+0000 7fdfc6257640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:50.909 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.921+0000 7fdfbe7fc640 1 -- 172.21.15.57:0/2343472748 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdfb0001720 con 0x7fdfc00a9d70 2024-03-29T10:41:50.910 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.921+0000 7fdfbe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:50.910 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.921+0000 7fdfbe7fc640 1 -- 172.21.15.57:0/2343472748 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fdfb0001b30 con 0x7fdfc00a9d70 2024-03-29T10:41:50.910 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.921+0000 7fdfbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:50.910 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.922+0000 7fdfbe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:50.910 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.922+0000 7fdfbe7fc640 10 monclient: dump: 2024-03-29T10:41:50.910 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:50.910 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:50.910 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:50.910 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:50.910 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:50.910 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:50.910 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:50.910 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:50.910 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:50.911 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:50.911 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.922+0000 7fdfc6257640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:50.911 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.922+0000 7fdfc6257640 4 set_mon_vals no callback set 2024-03-29T10:41:50.911 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.923+0000 7fdfc6257640 10 monclient: shutdown 2024-03-29T10:41:50.911 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.923+0000 7fdfc6257640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:50.911 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.923+0000 7fdfc6257640 1 -- 172.21.15.57:0/2343472748 >> v1:172.21.15.179:6789/0 conn(0x7fdfc00a9d70 legacy=0x7fdfc014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:50.911 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.923+0000 7fdfc6257640 1 -- 172.21.15.57:0/2343472748 shutdown_connections 2024-03-29T10:41:50.911 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.923+0000 7fdfc6257640 1 -- 172.21.15.57:0/2343472748 >> 172.21.15.57:0/2343472748 conn(0x7fdfc0066a00 msgr2=0x7fdfc0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:50.911 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.923+0000 7fdfc6257640 1 -- 172.21.15.57:0/2343472748 shutdown_connections 2024-03-29T10:41:50.916 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.923+0000 7fdfc6257640 1 -- 172.21.15.57:0/2343472748 wait complete. 2024-03-29T10:41:50.916 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.924+0000 7fdfc6257640 10 monclient: build_initial_monmap 2024-03-29T10:41:50.916 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.924+0000 7fdfc6257640 10 monclient: monmap: 2024-03-29T10:41:50.916 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:41:50.917 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:50.917 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:41:50.917 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:41:50.917 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:50.917 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:50.917 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:41:50.917 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:41:50.917 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:41:50.917 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:50.917 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.925+0000 7fdfc6257640 1 librados: starting msgr at 2024-03-29T10:41:50.917 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.925+0000 7fdfc6257640 1 librados: starting objecter 2024-03-29T10:41:50.917 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.925+0000 7fdfc6257640 1 Processor -- start 2024-03-29T10:41:50.917 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.925+0000 7fdfc6257640 1 -- start start 2024-03-29T10:41:50.917 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.925+0000 7fdfc6257640 1 librados: setting wanted keys 2024-03-29T10:41:50.917 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.925+0000 7fdfc6257640 1 librados: calling monclient init 2024-03-29T10:41:50.918 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.925+0000 7fdfc6257640 10 monclient: init 2024-03-29T10:41:50.918 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.926+0000 7fdfc6257640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:50.918 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.926+0000 7fdfc6257640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:41:50.918 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.926+0000 7fdfc6257640 10 monclient(hunting): picked mon.noname-c con 0x7fdfc00a9d70 addr v1:172.21.15.57:6790/0 2024-03-29T10:41:50.918 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.927+0000 7fdfc6257640 10 monclient(hunting): picked mon.noname-a con 0x7fdfc013f650 addr v1:172.21.15.57:6789/0 2024-03-29T10:41:50.918 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.927+0000 7fdfc6257640 10 monclient(hunting): picked mon.noname-b con 0x7fdfc01457f0 addr v1:172.21.15.179:6789/0 2024-03-29T10:41:50.918 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.927+0000 7fdfc6257640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc006f990 con 0x7fdfc013f650 2024-03-29T10:41:50.918 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.927+0000 7fdfc6257640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc006fc70 con 0x7fdfc01457f0 2024-03-29T10:41:50.918 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.927+0000 7fdfbf7fe640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7fdfc00a9d70 0x7fdfc0137e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:54516/0 (socket says 172.21.15.57:54516) 2024-03-29T10:41:50.918 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.927+0000 7fdfbf7fe640 1 -- 172.21.15.57:0/1220460053 learned_addr learned my addr 172.21.15.57:0/1220460053 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:41:50.918 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.927+0000 7fdfc6257640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc006de90 con 0x7fdfc00a9d70 2024-03-29T10:41:50.918 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.927+0000 7fdfbffff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fdfc01457f0 0x7fdfc01dc0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:46468/0 (socket says 172.21.15.57:46468) 2024-03-29T10:41:50.918 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.927+0000 7fdfc6257640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:50.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.927+0000 7fdfa7fff640 10 client.?.objecter ms_handle_connect 0x7fdfc01457f0 2024-03-29T10:41:50.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.927+0000 7fdfa7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:50.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.927+0000 7fdfc6257640 10 monclient(hunting): authenticate will time out at 1063.103738s 2024-03-29T10:41:50.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.927+0000 7fdfa7fff640 10 client.?.objecter ms_handle_connect 0x7fdfc013f650 2024-03-29T10:41:50.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.927+0000 7fdfa7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:50.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.927+0000 7fdfa7fff640 10 client.?.objecter ms_handle_connect 0x7fdfc00a9d70 2024-03-29T10:41:50.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.927+0000 7fdfa7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:50.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 1 -- 172.21.15.57:0/1220460053 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660226520 0 0) 0x7fdfc006fc70 con 0x7fdfc01457f0 2024-03-29T10:41:50.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:50.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:50.920 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:50.920 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 10 monclient(hunting): my global_id is 4312 2024-03-29T10:41:50.920 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 1 -- 172.21.15.57:0/1220460053 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfac004eb0 con 0x7fdfc01457f0 2024-03-29T10:41:50.920 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 1 -- 172.21.15.57:0/1220460053 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660785221 0 0) 0x7fdfc006f990 con 0x7fdfc013f650 2024-03-29T10:41:50.920 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:50.921 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:50.921 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:50.921 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 10 monclient(hunting): my global_id is 4305 2024-03-29T10:41:50.921 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 1 -- 172.21.15.57:0/1220460053 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfc006fc70 con 0x7fdfc013f650 2024-03-29T10:41:50.921 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 1 -- 172.21.15.57:0/1220460053 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2215882394 0 0) 0x7fdfac004eb0 con 0x7fdfc01457f0 2024-03-29T10:41:50.921 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 1 -- 172.21.15.57:0/1220460053 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfc006f990 con 0x7fdfc01457f0 2024-03-29T10:41:50.921 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 1 -- 172.21.15.57:0/1220460053 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fdfb0001a30 con 0x7fdfc01457f0 2024-03-29T10:41:50.921 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:50.921 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:50.921 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 10 monclient(hunting): dump: 2024-03-29T10:41:50.921 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:50.922 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:50.922 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:50.922 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:50.922 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:50.922 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:50.922 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:50.922 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:50.922 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:50.922 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:50.922 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:50.922 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 1 -- 172.21.15.57:0/1220460053 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562063941 0 0) 0x7fdfc006de90 con 0x7fdfc00a9d70 2024-03-29T10:41:50.923 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:50.923 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:50.923 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:50.923 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 10 monclient(hunting): my global_id is 4316 2024-03-29T10:41:50.923 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.928+0000 7fdfa7fff640 1 -- 172.21.15.57:0/1220460053 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfac004eb0 con 0x7fdfc00a9d70 2024-03-29T10:41:50.923 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfa7fff640 1 -- 172.21.15.57:0/1220460053 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 172745508 0 0) 0x7fdfac004eb0 con 0x7fdfc00a9d70 2024-03-29T10:41:50.923 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfa7fff640 1 -- 172.21.15.57:0/1220460053 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfc006de90 con 0x7fdfc00a9d70 2024-03-29T10:41:50.923 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfa7fff640 1 -- 172.21.15.57:0/1220460053 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fdfb4001780 con 0x7fdfc00a9d70 2024-03-29T10:41:50.923 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfa7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:50.923 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfa7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:50.923 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfa7fff640 10 monclient(hunting): dump: 2024-03-29T10:41:50.923 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:50.924 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:50.924 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:50.924 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:50.924 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:50.924 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:50.924 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:50.924 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:50.924 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:50.924 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:50.924 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfa7fff640 1 -- 172.21.15.57:0/1220460053 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2391812883 0 0) 0x7fdfc006f990 con 0x7fdfc01457f0 2024-03-29T10:41:50.924 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfa7fff640 1 -- 172.21.15.57:0/1220460053 >> v1:172.21.15.57:6790/0 conn(0x7fdfc00a9d70 legacy=0x7fdfc0137e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:50.924 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfa7fff640 1 -- 172.21.15.57:0/1220460053 >> v1:172.21.15.57:6789/0 conn(0x7fdfc013f650 legacy=0x7fdfc01db9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:50.925 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfa7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:41:50.925 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfa7fff640 1 monclient: found mon.b 2024-03-29T10:41:50.925 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfa7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:50.925 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfa7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:50.925 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfa7fff640 1 -- 172.21.15.57:0/1220460053 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc0138ed0 con 0x7fdfc01457f0 2024-03-29T10:41:50.925 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfa7fff640 10 monclient: _finish_auth 0 2024-03-29T10:41:50.925 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfa7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:41:50.925 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:50.925 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfc6257640 5 monclient: authenticate success, global_id 4312 2024-03-29T10:41:50.925 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfc6257640 10 monclient: _renew_subs 2024-03-29T10:41:50.925 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfc6257640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:50.925 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfc6257640 1 -- 172.21.15.57:0/1220460053 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfc0138570 con 0x7fdfc01457f0 2024-03-29T10:41:50.925 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfc6257640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:50.926 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfc6257640 10 monclient: _renew_subs 2024-03-29T10:41:50.926 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfc6257640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:50.926 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfc6257640 1 -- 172.21.15.57:0/1220460053 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfc0138870 con 0x7fdfc01457f0 2024-03-29T10:41:50.926 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.929+0000 7fdfc6257640 1 librados: init done 2024-03-29T10:41:50.926 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.930+0000 7fdfa7fff640 1 -- 172.21.15.57:0/1220460053 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdfb00038a0 con 0x7fdfc01457f0 2024-03-29T10:41:50.926 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.930+0000 7fdfa7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:50.926 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.930+0000 7fdfa7fff640 1 -- 172.21.15.57:0/1220460053 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fdfb0001180 con 0x7fdfc01457f0 2024-03-29T10:41:50.926 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.930+0000 7fdfa7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:50.926 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.930+0000 7fdfa7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:50.926 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.930+0000 7fdfa7fff640 10 monclient: dump: 2024-03-29T10:41:50.926 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:50.926 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:50.927 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:50.927 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:50.927 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:50.927 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:50.927 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:50.927 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:50.927 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:50.927 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:50.927 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.931+0000 7fdfbd7fa640 4 set_mon_vals no callback set 2024-03-29T10:41:50.927 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.932+0000 7fdfa7fff640 1 -- 172.21.15.57:0/1220460053 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fdfb0003d30 con 0x7fdfc01457f0 2024-03-29T10:41:50.927 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.932+0000 7fdfc6257640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:50.927 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.932+0000 7fdfc6257640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:50.927 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.932+0000 7fdfc6257640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:50.928 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.932+0000 7fdfc6257640 1 -- 172.21.15.57:0/1220460053 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa0005080 con 0x7fdfc01457f0 2024-03-29T10:41:50.928 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.933+0000 7fdfa7fff640 1 -- 172.21.15.57:0/1220460053 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7fdfb0084a50 con 0x7fdfc01457f0 2024-03-29T10:41:50.928 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.933+0000 7fdfbf7fe640 10 monclient: get_auth_request con 0x7fdfac06de50 auth_method 0 2024-03-29T10:41:50.928 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.933+0000 7fdfa7fff640 10 client.4312.objecter ms_dispatch 0x7fdfc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:50.928 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.933+0000 7fdfa7fff640 3 client.4312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:50.928 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.933+0000 7fdfa7fff640 3 client.4312.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:50.928 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.933+0000 7fdfa7fff640 20 client.4312.objecter dump_active .. 0 homeless 2024-03-29T10:41:50.929 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.934+0000 7fdfa7fff640 10 client.4312.objecter ms_handle_connect 0x7fdfac06de50 2024-03-29T10:41:50.929 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.939+0000 7fdfa7fff640 1 -- 172.21.15.57:0/1220460053 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdfb0050330 con 0x7fdfc01457f0 2024-03-29T10:41:50.929 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.940+0000 7fdfa7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:50.929 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:50.940+0000 7fdfa7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:51.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:51.343+0000 7fdfc6257640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:51.327 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:51.343+0000 7fdfc6257640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:51.327 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:51.343+0000 7fdfc6257640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:41:51.327 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:51.343+0000 7fdfc6257640 1 -- 172.21.15.57:0/1220460053 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdfa0005330 con 0x7fdfc01457f0 2024-03-29T10:41:51.327 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:51.344+0000 7fdfa7fff640 1 -- 172.21.15.57:0/1220460053 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7fdfb0052260 con 0x7fdfc01457f0 2024-03-29T10:41:51.327 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:51.344+0000 7fdfa7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:51.328 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:51.344+0000 7fdfa7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:51.328 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:41:51.328 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:51.331 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:51.348+0000 7fdfc6257640 10 librados: watch_flush enter 2024-03-29T10:41:51.331 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:51.348+0000 7fdfc6257640 10 librados: watch_flush exit 2024-03-29T10:41:51.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:51.348+0000 7fdfc6257640 20 client.4312.objecter shutdown clearing up homeless session... 2024-03-29T10:41:51.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:51.348+0000 7fdfc6257640 10 client.4312.objecter successfully canceled tick 2024-03-29T10:41:51.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:51.348+0000 7fdfc6257640 1 -- 172.21.15.57:0/1220460053 >> v1:172.21.15.179:6816/2698295936 conn(0x7fdfac06de50 legacy=0x7fdfac070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:51.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:51.348+0000 7fdfc6257640 10 monclient: shutdown 2024-03-29T10:41:51.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:51.348+0000 7fdfc6257640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:51.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:51.348+0000 7fdfc6257640 1 -- 172.21.15.57:0/1220460053 >> v1:172.21.15.179:6789/0 conn(0x7fdfc01457f0 legacy=0x7fdfc01dc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:51.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:51.348+0000 7fdfc6257640 1 -- 172.21.15.57:0/1220460053 shutdown_connections 2024-03-29T10:41:51.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:51.348+0000 7fdfc6257640 1 -- 172.21.15.57:0/1220460053 >> 172.21.15.57:0/1220460053 conn(0x7fdfc0066a00 msgr2=0x7fdfc013cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:51.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:51.348+0000 7fdfc6257640 1 -- 172.21.15.57:0/1220460053 shutdown_connections 2024-03-29T10:41:51.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:51.348+0000 7fdfc6257640 1 -- 172.21.15.57:0/1220460053 wait complete. 2024-03-29T10:41:51.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:51.348+0000 7fdfc6257640 1 librados: shutdown 2024-03-29T10:41:51.345 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:58.153 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:58.303 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.320+0000 7feb9884e640 10 monclient: get_monmap_and_config 2024-03-29T10:41:58.303 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.320+0000 7feb9884e640 10 monclient: build_initial_monmap 2024-03-29T10:41:58.304 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.320+0000 7feb9884e640 10 monclient: monmap: 2024-03-29T10:41:58.304 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:41:58.304 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:58.304 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:41:58.320925+0000 2024-03-29T10:41:58.304 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:41:58.320925+0000 2024-03-29T10:41:58.304 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:58.304 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:58.304 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:41:58.304 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:41:58.304 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:41:58.304 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:58.306 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.321+0000 7feb9884e640 1 Processor -- start 2024-03-29T10:41:58.306 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.321+0000 7feb9884e640 1 -- start start 2024-03-29T10:41:58.306 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.321+0000 7feb9884e640 10 monclient: init 2024-03-29T10:41:58.306 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.322+0000 7feb9884e640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:58.306 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.322+0000 7feb9884e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:41:58.306 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.322+0000 7feb9884e640 10 monclient(hunting): picked mon.noname-a con 0x7feb941457f0 addr v1:172.21.15.57:6789/0 2024-03-29T10:41:58.306 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.322+0000 7feb9259c640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7feb941457f0 0x7feb9413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:53868/0 (socket says 172.21.15.57:53868) 2024-03-29T10:41:58.306 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.322+0000 7feb9884e640 10 monclient(hunting): picked mon.noname-b con 0x7feb9413f650 addr v1:172.21.15.57:6790/0 2024-03-29T10:41:58.306 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.322+0000 7feb9884e640 10 monclient(hunting): picked mon.noname-c con 0x7feb940a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T10:41:58.306 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.322+0000 7feb9884e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb940aaef0 con 0x7feb941457f0 2024-03-29T10:41:58.307 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.322+0000 7feb9259c640 1 -- 172.21.15.57:0/2018109490 learned_addr learned my addr 172.21.15.57:0/2018109490 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:41:58.307 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.323+0000 7feb9884e640 1 -- 172.21.15.57:0/2018109490 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb94145bd0 con 0x7feb940a9d70 2024-03-29T10:41:58.307 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.323+0000 7feb9884e640 1 -- 172.21.15.57:0/2018109490 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb94145eb0 con 0x7feb9413f650 2024-03-29T10:41:58.307 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.323+0000 7feb9884e640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:58.307 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.323+0000 7feb9884e640 10 monclient(hunting): authenticate will time out at 1070.499223s 2024-03-29T10:41:58.308 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.323+0000 7feb9159a640 1 -- 172.21.15.57:0/2018109490 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225517915 0 0) 0x7feb94145eb0 con 0x7feb9413f650 2024-03-29T10:41:58.308 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 10 monclient(hunting): _negotiate 2024-03-29T10:41:58.308 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:58.308 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:58.308 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 10 monclient(hunting): my global_id is 4319 2024-03-29T10:41:58.308 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 1 -- 172.21.15.57:0/2018109490 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb7c003520 con 0x7feb9413f650 2024-03-29T10:41:58.308 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 1 -- 172.21.15.57:0/2018109490 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810674371 0 0) 0x7feb940aaef0 con 0x7feb941457f0 2024-03-29T10:41:58.308 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 10 monclient(hunting): _negotiate 2024-03-29T10:41:58.308 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:58.309 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:58.309 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 10 monclient(hunting): my global_id is 4308 2024-03-29T10:41:58.309 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 1 -- 172.21.15.57:0/2018109490 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb94145eb0 con 0x7feb941457f0 2024-03-29T10:41:58.309 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 1 -- 172.21.15.57:0/2018109490 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 960312462 0 0) 0x7feb7c003520 con 0x7feb9413f650 2024-03-29T10:41:58.309 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 1 -- 172.21.15.57:0/2018109490 >> v1:172.21.15.179:6789/0 conn(0x7feb940a9d70 legacy=0x7feb9414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:58.309 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 1 -- 172.21.15.57:0/2018109490 >> v1:172.21.15.57:6789/0 conn(0x7feb941457f0 legacy=0x7feb9413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:58.309 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 10 monclient: _finish_hunting 0 2024-03-29T10:41:58.309 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 1 monclient: found mon.noname-b 2024-03-29T10:41:58.309 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:58.309 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:41:58.309 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 1 -- 172.21.15.57:0/2018109490 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb94146340 con 0x7feb9413f650 2024-03-29T10:41:58.309 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 10 monclient: _finish_auth 0 2024-03-29T10:41:58.309 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 10 monclient: _check_auth_tickets 2024-03-29T10:41:58.310 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:58.310 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 1 -- 172.21.15.57:0/2018109490 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7feb88001690 con 0x7feb9413f650 2024-03-29T10:41:58.310 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:58.310 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:58.310 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9159a640 10 monclient: dump: 2024-03-29T10:41:58.310 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:58.310 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:58.310 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:58.310 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:58.310 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:58.310 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:58.310 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:58.310 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:58.310 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:58.311 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:58.311 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9884e640 5 monclient: authenticate success, global_id 4319 2024-03-29T10:41:58.311 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.324+0000 7feb9884e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:58.311 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.325+0000 7feb9159a640 1 -- 172.21.15.57:0/2018109490 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb880017f0 con 0x7feb9413f650 2024-03-29T10:41:58.311 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.325+0000 7feb9159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:58.311 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.325+0000 7feb9159a640 1 -- 172.21.15.57:0/2018109490 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7feb88001c00 con 0x7feb9413f650 2024-03-29T10:41:58.311 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.325+0000 7feb9159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:58.311 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.325+0000 7feb9159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:58.311 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.325+0000 7feb9159a640 10 monclient: dump: 2024-03-29T10:41:58.311 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:58.311 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:58.311 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:58.311 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:58.311 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:58.312 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:58.312 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:58.312 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:58.312 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:58.312 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:58.312 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.325+0000 7feb9884e640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:58.312 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.325+0000 7feb9884e640 4 set_mon_vals no callback set 2024-03-29T10:41:58.319 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.326+0000 7feb9884e640 10 monclient: shutdown 2024-03-29T10:41:58.319 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.326+0000 7feb9884e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:58.319 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.326+0000 7feb9884e640 1 -- 172.21.15.57:0/2018109490 >> v1:172.21.15.57:6790/0 conn(0x7feb9413f650 legacy=0x7feb940b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:58.319 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.326+0000 7feb9884e640 1 -- 172.21.15.57:0/2018109490 shutdown_connections 2024-03-29T10:41:58.319 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.326+0000 7feb9884e640 1 -- 172.21.15.57:0/2018109490 >> 172.21.15.57:0/2018109490 conn(0x7feb94066a00 msgr2=0x7feb94066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:58.319 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.326+0000 7feb9884e640 1 -- 172.21.15.57:0/2018109490 shutdown_connections 2024-03-29T10:41:58.319 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.326+0000 7feb9884e640 1 -- 172.21.15.57:0/2018109490 wait complete. 2024-03-29T10:41:58.319 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.328+0000 7feb9884e640 10 monclient: build_initial_monmap 2024-03-29T10:41:58.320 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.328+0000 7feb9884e640 10 monclient: monmap: 2024-03-29T10:41:58.320 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:41:58.320 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:58.320 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:41:58.320 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:41:58.320 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:58.320 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:58.320 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:41:58.320 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:41:58.320 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:41:58.320 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:58.320 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.328+0000 7feb9884e640 1 librados: starting msgr at 2024-03-29T10:41:58.320 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.328+0000 7feb9884e640 1 librados: starting objecter 2024-03-29T10:41:58.320 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.328+0000 7feb9884e640 1 Processor -- start 2024-03-29T10:41:58.320 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.329+0000 7feb9884e640 1 -- start start 2024-03-29T10:41:58.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.329+0000 7feb9884e640 1 librados: setting wanted keys 2024-03-29T10:41:58.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.329+0000 7feb9884e640 1 librados: calling monclient init 2024-03-29T10:41:58.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.329+0000 7feb9884e640 10 monclient: init 2024-03-29T10:41:58.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.330+0000 7feb9884e640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:58.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.330+0000 7feb9884e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:58.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.330+0000 7feb9884e640 10 monclient(hunting): picked mon.noname-b con 0x7feb940a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T10:41:58.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.330+0000 7feb9884e640 10 monclient(hunting): picked mon.noname-a con 0x7feb9413f650 addr v1:172.21.15.57:6789/0 2024-03-29T10:41:58.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.330+0000 7feb9884e640 10 monclient(hunting): picked mon.noname-c con 0x7feb941457f0 addr v1:172.21.15.57:6790/0 2024-03-29T10:41:58.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.330+0000 7feb9884e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb94142a00 con 0x7feb9413f650 2024-03-29T10:41:58.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.330+0000 7feb9884e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb941425e0 con 0x7feb940a9d70 2024-03-29T10:41:58.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.330+0000 7feb9884e640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb94142c30 con 0x7feb941457f0 2024-03-29T10:41:58.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.330+0000 7feb9884e640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:58.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.330+0000 7feb9884e640 10 monclient(hunting): authenticate will time out at 1070.506717s 2024-03-29T10:41:58.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.331+0000 7feb91d9b640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7feb9413f650 0x7feb94134fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:54306/0 (socket says 172.21.15.57:54306) 2024-03-29T10:41:58.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.331+0000 7feb91d9b640 1 -- 172.21.15.57:0/2543222020 learned_addr learned my addr 172.21.15.57:0/2543222020 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:41:58.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.331+0000 7feb82ffd640 10 client.?.objecter ms_handle_connect 0x7feb9413f650 2024-03-29T10:41:58.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.331+0000 7feb82ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:58.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.331+0000 7feb82ffd640 1 -- 172.21.15.57:0/2543222020 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4007405610 0 0) 0x7feb94142a00 con 0x7feb9413f650 2024-03-29T10:41:58.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.331+0000 7feb82ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:58.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.331+0000 7feb82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:58.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.331+0000 7feb82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:58.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.331+0000 7feb82ffd640 10 monclient(hunting): my global_id is 4311 2024-03-29T10:41:58.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.331+0000 7feb82ffd640 1 -- 172.21.15.57:0/2543222020 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb74004eb0 con 0x7feb9413f650 2024-03-29T10:41:58.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.331+0000 7feb82ffd640 10 client.?.objecter ms_handle_connect 0x7feb941457f0 2024-03-29T10:41:58.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.331+0000 7feb82ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:58.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.332+0000 7feb82ffd640 10 client.?.objecter ms_handle_connect 0x7feb940a9d70 2024-03-29T10:41:58.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.332+0000 7feb82ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:58.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.332+0000 7feb82ffd640 1 -- 172.21.15.57:0/2543222020 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989782254 0 0) 0x7feb94142c30 con 0x7feb941457f0 2024-03-29T10:41:58.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.332+0000 7feb82ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:58.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.332+0000 7feb82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:58.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.332+0000 7feb82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:58.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.332+0000 7feb82ffd640 10 monclient(hunting): my global_id is 4322 2024-03-29T10:41:58.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.332+0000 7feb82ffd640 1 -- 172.21.15.57:0/2543222020 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb94142a00 con 0x7feb941457f0 2024-03-29T10:41:58.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.332+0000 7feb82ffd640 1 -- 172.21.15.57:0/2543222020 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 583739184 0 0) 0x7feb74004eb0 con 0x7feb9413f650 2024-03-29T10:41:58.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.332+0000 7feb82ffd640 1 -- 172.21.15.57:0/2543222020 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb94142c30 con 0x7feb9413f650 2024-03-29T10:41:58.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.332+0000 7feb82ffd640 1 -- 172.21.15.57:0/2543222020 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7feb88001bc0 con 0x7feb9413f650 2024-03-29T10:41:58.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.332+0000 7feb82ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:58.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.332+0000 7feb82ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:58.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.332+0000 7feb82ffd640 10 monclient(hunting): dump: 2024-03-29T10:41:58.323 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:58.324 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:58.324 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:58.324 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:58.324 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:58.324 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:58.324 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:58.324 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:58.324 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:58.324 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:58.324 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.332+0000 7feb82ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:58.324 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.332+0000 7feb82ffd640 1 -- 172.21.15.57:0/2543222020 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 749059204 0 0) 0x7feb941425e0 con 0x7feb940a9d70 2024-03-29T10:41:58.324 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.332+0000 7feb82ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:58.324 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.332+0000 7feb82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:58.324 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.332+0000 7feb82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:58.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.332+0000 7feb82ffd640 10 monclient(hunting): my global_id is 4318 2024-03-29T10:41:58.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.332+0000 7feb82ffd640 1 -- 172.21.15.57:0/2543222020 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb74004eb0 con 0x7feb940a9d70 2024-03-29T10:41:58.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb82ffd640 1 -- 172.21.15.57:0/2543222020 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3037578765 0 0) 0x7feb94142c30 con 0x7feb9413f650 2024-03-29T10:41:58.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb82ffd640 1 -- 172.21.15.57:0/2543222020 >> v1:172.21.15.57:6790/0 conn(0x7feb941457f0 legacy=0x7feb941daa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:58.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb82ffd640 1 -- 172.21.15.57:0/2543222020 >> v1:172.21.15.179:6789/0 conn(0x7feb940a9d70 legacy=0x7feb94140c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:58.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb82ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:41:58.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb82ffd640 1 monclient: found mon.a 2024-03-29T10:41:58.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:58.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb82ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:41:58.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb82ffd640 1 -- 172.21.15.57:0/2543222020 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb941413a0 con 0x7feb9413f650 2024-03-29T10:41:58.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb82ffd640 10 monclient: _finish_auth 0 2024-03-29T10:41:58.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb82ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:41:58.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:58.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb9884e640 5 monclient: authenticate success, global_id 4311 2024-03-29T10:41:58.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb9884e640 10 monclient: _renew_subs 2024-03-29T10:41:58.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb9884e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:41:58.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb9884e640 1 -- 172.21.15.57:0/2543222020 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb9413fda0 con 0x7feb9413f650 2024-03-29T10:41:58.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb9884e640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:58.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb9884e640 10 monclient: _renew_subs 2024-03-29T10:41:58.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb9884e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:41:58.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb9884e640 1 -- 172.21.15.57:0/2543222020 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb941358c0 con 0x7feb9413f650 2024-03-29T10:41:58.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb9884e640 1 librados: init done 2024-03-29T10:41:58.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb82ffd640 1 -- 172.21.15.57:0/2543222020 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb880011f0 con 0x7feb9413f650 2024-03-29T10:41:58.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb82ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:58.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb82ffd640 1 -- 172.21.15.57:0/2543222020 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7feb88003b00 con 0x7feb9413f650 2024-03-29T10:41:58.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:58.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb82ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:58.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.333+0000 7feb82ffd640 10 monclient: dump: 2024-03-29T10:41:58.326 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:41:58.327 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:41:58.327 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:58.327 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:41:58.327 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:58.327 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:41:58.327 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:41:58.327 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:41:58.327 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:41:58.327 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:41:58.327 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.334+0000 7feb90d99640 4 set_mon_vals no callback set 2024-03-29T10:41:58.327 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.334+0000 7feb9884e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:58.327 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.334+0000 7feb9884e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:58.327 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.335+0000 7feb9884e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:41:58.327 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.335+0000 7feb9884e640 1 -- 172.21.15.57:0/2543222020 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb78005080 con 0x7feb9413f650 2024-03-29T10:41:58.327 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.336+0000 7feb82ffd640 1 -- 172.21.15.57:0/2543222020 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7feb88003c60 con 0x7feb9413f650 2024-03-29T10:41:58.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.337+0000 7feb9259c640 10 monclient: get_auth_request con 0x7feb7406de50 auth_method 0 2024-03-29T10:41:58.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.337+0000 7feb82ffd640 1 -- 172.21.15.57:0/2543222020 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7feb88084b80 con 0x7feb9413f650 2024-03-29T10:41:58.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.337+0000 7feb82ffd640 10 client.4311.objecter ms_dispatch 0x7feb94000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:58.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.337+0000 7feb82ffd640 3 client.4311.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:58.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.337+0000 7feb82ffd640 3 client.4311.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:58.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.337+0000 7feb82ffd640 20 client.4311.objecter dump_active .. 0 homeless 2024-03-29T10:41:58.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.337+0000 7feb82ffd640 10 client.4311.objecter ms_handle_connect 0x7feb7406de50 2024-03-29T10:41:58.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.346+0000 7feb82ffd640 1 -- 172.21.15.57:0/2543222020 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feb88050290 con 0x7feb9413f650 2024-03-29T10:41:58.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.346+0000 7feb82ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:58.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.346+0000 7feb82ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:58.701 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.718+0000 7feb9884e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:58.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.718+0000 7feb9884e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:58.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.718+0000 7feb9884e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:41:58.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.718+0000 7feb9884e640 1 -- 172.21.15.57:0/2543222020 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb78005330 con 0x7feb9413f650 2024-03-29T10:41:58.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.719+0000 7feb82ffd640 1 -- 172.21.15.57:0/2543222020 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7feb88052390 con 0x7feb9413f650 2024-03-29T10:41:58.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.719+0000 7feb82ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:58.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.719+0000 7feb82ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:58.703 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:41:58.703 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:58.706 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.723+0000 7feb9884e640 10 librados: watch_flush enter 2024-03-29T10:41:58.706 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.723+0000 7feb9884e640 10 librados: watch_flush exit 2024-03-29T10:41:58.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.723+0000 7feb9884e640 20 client.4311.objecter shutdown clearing up homeless session... 2024-03-29T10:41:58.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.723+0000 7feb9884e640 10 client.4311.objecter successfully canceled tick 2024-03-29T10:41:58.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.723+0000 7feb9884e640 1 -- 172.21.15.57:0/2543222020 >> v1:172.21.15.179:6816/2698295936 conn(0x7feb7406de50 legacy=0x7feb74070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:58.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.723+0000 7feb9884e640 10 monclient: shutdown 2024-03-29T10:41:58.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.723+0000 7feb9884e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:58.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.723+0000 7feb9884e640 1 -- 172.21.15.57:0/2543222020 >> v1:172.21.15.57:6789/0 conn(0x7feb9413f650 legacy=0x7feb94134fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:58.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.723+0000 7feb9884e640 1 -- 172.21.15.57:0/2543222020 shutdown_connections 2024-03-29T10:41:58.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.723+0000 7feb9884e640 1 -- 172.21.15.57:0/2543222020 >> 172.21.15.57:0/2543222020 conn(0x7feb94066a00 msgr2=0x7feb940b3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:58.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.723+0000 7feb9884e640 1 -- 172.21.15.57:0/2543222020 shutdown_connections 2024-03-29T10:41:58.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.723+0000 7feb9884e640 1 -- 172.21.15.57:0/2543222020 wait complete. 2024-03-29T10:41:58.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:41:58.724+0000 7feb9884e640 1 librados: shutdown 2024-03-29T10:41:58.720 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:05.527 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:05.675 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.690+0000 7f065c801640 10 monclient: get_monmap_and_config 2024-03-29T10:42:05.675 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.690+0000 7f065c801640 10 monclient: build_initial_monmap 2024-03-29T10:42:05.675 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.690+0000 7f065c801640 10 monclient: monmap: 2024-03-29T10:42:05.675 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:42:05.675 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:05.675 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:42:05.692345+0000 2024-03-29T10:42:05.675 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:42:05.692345+0000 2024-03-29T10:42:05.675 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:05.676 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:05.676 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:42:05.676 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:42:05.676 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:42:05.676 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:05.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.691+0000 7f065c801640 1 Processor -- start 2024-03-29T10:42:05.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.692+0000 7f065c801640 1 -- start start 2024-03-29T10:42:05.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.692+0000 7f065c801640 10 monclient: init 2024-03-29T10:42:05.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.692+0000 7f065c801640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:05.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.692+0000 7f065c801640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:42:05.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.693+0000 7f065c801640 10 monclient(hunting): picked mon.noname-b con 0x7f0650006a70 addr v1:172.21.15.57:6790/0 2024-03-29T10:42:05.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.693+0000 7f065c801640 10 monclient(hunting): picked mon.noname-a con 0x7f065002b100 addr v1:172.21.15.57:6789/0 2024-03-29T10:42:05.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.693+0000 7f065c801640 10 monclient(hunting): picked mon.noname-c con 0x7f0650029c30 addr v1:172.21.15.179:6789/0 2024-03-29T10:42:05.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.693+0000 7f065c801640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f065002a270 con 0x7f065002b100 2024-03-29T10:42:05.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.693+0000 7f065c801640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f065002c440 con 0x7f0650029c30 2024-03-29T10:42:05.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.693+0000 7f06577fe640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f0650006a70 0x7f06500c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:59130/0 (socket says 172.21.15.57:59130) 2024-03-29T10:42:05.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.693+0000 7f06577fe640 1 -- 172.21.15.57:0/2046691432 learned_addr learned my addr 172.21.15.57:0/2046691432 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:42:05.679 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.693+0000 7f065c801640 1 -- 172.21.15.57:0/2046691432 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06500c9440 con 0x7f0650006a70 2024-03-29T10:42:05.679 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.693+0000 7f065c801640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:05.679 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.693+0000 7f065c801640 10 monclient(hunting): authenticate will time out at 1077.870383s 2024-03-29T10:42:05.680 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.697+0000 7f06567fc640 1 -- 172.21.15.57:0/2046691432 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637065202 0 0) 0x7f06500c9440 con 0x7f0650006a70 2024-03-29T10:42:05.681 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.697+0000 7f06567fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:05.681 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.697+0000 7f06567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:05.681 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.697+0000 7f06567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:05.681 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.697+0000 7f06567fc640 10 monclient(hunting): my global_id is 4325 2024-03-29T10:42:05.681 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.697+0000 7f06567fc640 1 -- 172.21.15.57:0/2046691432 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0648003520 con 0x7f0650006a70 2024-03-29T10:42:05.681 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.697+0000 7f06567fc640 1 -- 172.21.15.57:0/2046691432 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3898870259 0 0) 0x7f0648003520 con 0x7f0650006a70 2024-03-29T10:42:05.682 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.697+0000 7f06567fc640 1 -- 172.21.15.57:0/2046691432 >> v1:172.21.15.179:6789/0 conn(0x7f0650029c30 legacy=0x7f06500d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:42:05.682 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.697+0000 7f06567fc640 1 -- 172.21.15.57:0/2046691432 >> v1:172.21.15.57:6789/0 conn(0x7f065002b100 legacy=0x7f06500cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:42:05.682 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.697+0000 7f06567fc640 10 monclient: _finish_hunting 0 2024-03-29T10:42:05.682 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.697+0000 7f06567fc640 1 monclient: found mon.noname-b 2024-03-29T10:42:05.682 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.697+0000 7f06567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:05.682 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.697+0000 7f06567fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:42:05.682 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.697+0000 7f06567fc640 1 -- 172.21.15.57:0/2046691432 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06500c95d0 con 0x7f0650006a70 2024-03-29T10:42:05.682 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.697+0000 7f06567fc640 10 monclient: _finish_auth 0 2024-03-29T10:42:05.682 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.697+0000 7f06567fc640 10 monclient: _check_auth_tickets 2024-03-29T10:42:05.682 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.697+0000 7f06567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:05.682 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.697+0000 7f06567fc640 1 -- 172.21.15.57:0/2046691432 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f0644001690 con 0x7f0650006a70 2024-03-29T10:42:05.683 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.697+0000 7f06567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:05.683 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.697+0000 7f06567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:05.683 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.697+0000 7f06567fc640 10 monclient: dump: 2024-03-29T10:42:05.683 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:05.683 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:05.683 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:05.683 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:05.683 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:05.683 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:05.683 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:05.683 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:05.683 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:05.683 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:05.683 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.697+0000 7f065c801640 5 monclient: authenticate success, global_id 4325 2024-03-29T10:42:05.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.697+0000 7f065c801640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:05.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.698+0000 7f06567fc640 1 -- 172.21.15.57:0/2046691432 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0644001860 con 0x7f0650006a70 2024-03-29T10:42:05.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.698+0000 7f06567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:05.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.698+0000 7f06567fc640 1 -- 172.21.15.57:0/2046691432 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f0644001cb0 con 0x7f0650006a70 2024-03-29T10:42:05.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.698+0000 7f06567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:05.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.698+0000 7f06567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:05.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.698+0000 7f06567fc640 10 monclient: dump: 2024-03-29T10:42:05.684 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:05.684 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:05.684 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:05.684 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:05.684 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:05.684 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:05.685 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:05.685 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:05.685 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:05.685 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:05.685 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.698+0000 7f065c801640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:05.685 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.698+0000 7f065c801640 4 set_mon_vals no callback set 2024-03-29T10:42:05.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.699+0000 7f065c801640 10 monclient: shutdown 2024-03-29T10:42:05.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.699+0000 7f065c801640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:05.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.699+0000 7f065c801640 1 -- 172.21.15.57:0/2046691432 >> v1:172.21.15.57:6790/0 conn(0x7f0650006a70 legacy=0x7f06500c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:05.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.699+0000 7f065c801640 1 -- 172.21.15.57:0/2046691432 shutdown_connections 2024-03-29T10:42:05.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.699+0000 7f065c801640 1 -- 172.21.15.57:0/2046691432 >> 172.21.15.57:0/2046691432 conn(0x7f06500058f0 msgr2=0x7f0650005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:05.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.699+0000 7f065c801640 1 -- 172.21.15.57:0/2046691432 shutdown_connections 2024-03-29T10:42:05.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.699+0000 7f065c801640 1 -- 172.21.15.57:0/2046691432 wait complete. 2024-03-29T10:42:05.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.700+0000 7f065c801640 10 monclient: build_initial_monmap 2024-03-29T10:42:05.688 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.700+0000 7f065c801640 10 monclient: monmap: 2024-03-29T10:42:05.688 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:42:05.688 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:05.688 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:42:05.688 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:42:05.688 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:05.688 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:05.688 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:42:05.688 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:42:05.688 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:42:05.688 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:05.688 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.701+0000 7f065c801640 1 librados: starting msgr at 2024-03-29T10:42:05.688 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.701+0000 7f065c801640 1 librados: starting objecter 2024-03-29T10:42:05.688 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.701+0000 7f065c801640 1 Processor -- start 2024-03-29T10:42:05.689 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.701+0000 7f065c801640 1 -- start start 2024-03-29T10:42:05.689 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.701+0000 7f065c801640 1 librados: setting wanted keys 2024-03-29T10:42:05.689 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.701+0000 7f065c801640 1 librados: calling monclient init 2024-03-29T10:42:05.689 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.701+0000 7f065c801640 10 monclient: init 2024-03-29T10:42:05.689 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.702+0000 7f065c801640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:05.689 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.702+0000 7f065c801640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:42:05.689 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.703+0000 7f065c801640 10 monclient(hunting): picked mon.noname-c con 0x7f0650006a70 addr v1:172.21.15.57:6790/0 2024-03-29T10:42:05.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.703+0000 7f065c801640 10 monclient(hunting): picked mon.noname-b con 0x7f0650029c30 addr v1:172.21.15.179:6789/0 2024-03-29T10:42:05.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.704+0000 7f065c801640 10 monclient(hunting): picked mon.noname-a con 0x7f06500c2e20 addr v1:172.21.15.57:6789/0 2024-03-29T10:42:05.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.705+0000 7f065c801640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f065002ef80 con 0x7f06500c2e20 2024-03-29T10:42:05.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.705+0000 7f065c801640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0650160880 con 0x7f0650029c30 2024-03-29T10:42:05.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.705+0000 7f065c801640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f065002f1d0 con 0x7f0650006a70 2024-03-29T10:42:05.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.705+0000 7f065c801640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:05.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.705+0000 7f065c801640 10 monclient(hunting): authenticate will time out at 1077.882145s 2024-03-29T10:42:05.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.705+0000 7f06577fe640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f0650006a70 0x7f06500c2710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:57530/0 (socket says 172.21.15.57:57530) 2024-03-29T10:42:05.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.705+0000 7f06577fe640 1 -- 172.21.15.57:0/2859802760 learned_addr learned my addr 172.21.15.57:0/2859802760 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:42:05.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.705+0000 7f0643fff640 10 client.?.objecter ms_handle_connect 0x7f0650006a70 2024-03-29T10:42:05.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.705+0000 7f0643fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:05.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.705+0000 7f0643fff640 1 -- 172.21.15.57:0/2859802760 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1881313549 0 0) 0x7f065002f1d0 con 0x7f0650006a70 2024-03-29T10:42:05.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.705+0000 7f0643fff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:05.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.705+0000 7f0643fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:05.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.706+0000 7f0643fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:05.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.706+0000 7f0643fff640 10 monclient(hunting): my global_id is 4328 2024-03-29T10:42:05.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.706+0000 7f0643fff640 1 -- 172.21.15.57:0/2859802760 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f063c004eb0 con 0x7f0650006a70 2024-03-29T10:42:05.694 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.706+0000 7f0643fff640 1 -- 172.21.15.57:0/2859802760 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3070034273 0 0) 0x7f063c004eb0 con 0x7f0650006a70 2024-03-29T10:42:05.694 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.706+0000 7f0643fff640 1 -- 172.21.15.57:0/2859802760 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f065002f1d0 con 0x7f0650006a70 2024-03-29T10:42:05.694 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.706+0000 7f0643fff640 1 -- 172.21.15.57:0/2859802760 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f06440019d0 con 0x7f0650006a70 2024-03-29T10:42:05.694 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.706+0000 7f0643fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:05.694 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.707+0000 7f0643fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:05.694 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.707+0000 7f0643fff640 10 monclient(hunting): dump: 2024-03-29T10:42:05.694 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:05.694 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:05.694 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:05.694 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:05.695 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:05.695 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:05.695 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:05.695 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:05.695 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:05.695 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:05.695 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.707+0000 7f0643fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:05.695 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.707+0000 7f0643fff640 10 client.?.objecter ms_handle_connect 0x7f0650029c30 2024-03-29T10:42:05.695 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.707+0000 7f0643fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:05.695 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.707+0000 7f0643fff640 1 -- 172.21.15.57:0/2859802760 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2742434873 0 0) 0x7f065002f1d0 con 0x7f0650006a70 2024-03-29T10:42:05.695 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.707+0000 7f0643fff640 1 -- 172.21.15.57:0/2859802760 >> v1:172.21.15.179:6789/0 conn(0x7f0650029c30 legacy=0x7f0650160d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:05.695 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.707+0000 7f0643fff640 1 -- 172.21.15.57:0/2859802760 >> v1:172.21.15.57:6789/0 conn(0x7f06500c2e20 legacy=0x7f065016c1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:05.695 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.708+0000 7f0643fff640 10 monclient: _finish_hunting 0 2024-03-29T10:42:05.695 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.708+0000 7f0643fff640 1 monclient: found mon.c 2024-03-29T10:42:05.696 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.708+0000 7f0643fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:05.696 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.708+0000 7f0643fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:42:05.696 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.708+0000 7f0643fff640 1 -- 172.21.15.57:0/2859802760 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06500cd080 con 0x7f0650006a70 2024-03-29T10:42:05.696 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.708+0000 7f0643fff640 10 monclient: _finish_auth 0 2024-03-29T10:42:05.696 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.708+0000 7f0643fff640 10 monclient: _check_auth_tickets 2024-03-29T10:42:05.696 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.708+0000 7f0643fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:05.696 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.708+0000 7f065c801640 5 monclient: authenticate success, global_id 4328 2024-03-29T10:42:05.696 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.708+0000 7f065c801640 10 monclient: _renew_subs 2024-03-29T10:42:05.696 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.708+0000 7f065c801640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:42:05.696 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.708+0000 7f065c801640 1 -- 172.21.15.57:0/2859802760 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f065015e3c0 con 0x7f0650006a70 2024-03-29T10:42:05.696 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.708+0000 7f065c801640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:05.696 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.708+0000 7f065c801640 10 monclient: _renew_subs 2024-03-29T10:42:05.696 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.708+0000 7f065c801640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:42:05.696 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.708+0000 7f065c801640 1 -- 172.21.15.57:0/2859802760 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0650161600 con 0x7f0650006a70 2024-03-29T10:42:05.696 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.709+0000 7f065c801640 1 librados: init done 2024-03-29T10:42:05.697 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.710+0000 7f065c801640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:05.697 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.710+0000 7f065c801640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:05.697 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.710+0000 7f0643fff640 1 -- 172.21.15.57:0/2859802760 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0644004bf0 con 0x7f0650006a70 2024-03-29T10:42:05.697 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.710+0000 7f065c801640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:42:05.697 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.710+0000 7f065c801640 1 -- 172.21.15.57:0/2859802760 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0650160a10 con 0x7f0650006a70 2024-03-29T10:42:05.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.713+0000 7f0643fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:05.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.713+0000 7f0643fff640 1 -- 172.21.15.57:0/2859802760 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f0644004f00 con 0x7f0650006a70 2024-03-29T10:42:05.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.713+0000 7f0643fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:05.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.713+0000 7f0643fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:05.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.713+0000 7f0643fff640 10 monclient: dump: 2024-03-29T10:42:05.699 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:05.699 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:05.700 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:05.700 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:05.700 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:05.700 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:05.700 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:05.700 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:05.700 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:05.700 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:05.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.713+0000 7f0643fff640 1 -- 172.21.15.57:0/2859802760 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f064401aed0 con 0x7f0650006a70 2024-03-29T10:42:05.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.714+0000 7f0643fff640 1 -- 172.21.15.57:0/2859802760 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f0644084e00 con 0x7f0650006a70 2024-03-29T10:42:05.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.714+0000 7f0643fff640 10 client.4328.objecter ms_dispatch 0x7f0658000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:05.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.714+0000 7f0643fff640 3 client.4328.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:05.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.714+0000 7f0643fff640 3 client.4328.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:05.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.714+0000 7f0643fff640 20 client.4328.objecter dump_active .. 0 homeless 2024-03-29T10:42:05.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.714+0000 7f06557fa640 4 set_mon_vals no callback set 2024-03-29T10:42:05.701 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.714+0000 7f0656ffd640 10 monclient: get_auth_request con 0x7f063c06db60 auth_method 0 2024-03-29T10:42:05.701 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.715+0000 7f0643fff640 10 client.4328.objecter ms_handle_connect 0x7f063c06db60 2024-03-29T10:42:05.708 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.724+0000 7f0643fff640 1 -- 172.21.15.57:0/2859802760 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0644050760 con 0x7f0650006a70 2024-03-29T10:42:05.708 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.724+0000 7f0643fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:05.708 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:05.724+0000 7f0643fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:06.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:06.059+0000 7f065c801640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:06.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:06.059+0000 7f065c801640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:06.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:06.059+0000 7f065c801640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:42:06.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:06.059+0000 7f065c801640 1 -- 172.21.15.57:0/2859802760 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0650004cd0 con 0x7f0650006a70 2024-03-29T10:42:06.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:06.063+0000 7f0643fff640 1 -- 172.21.15.57:0/2859802760 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f0644052690 con 0x7f0650006a70 2024-03-29T10:42:06.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:06.063+0000 7f0643fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:06.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:06.063+0000 7f0643fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:06.048 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:42:06.048 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:06.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:06.067+0000 7f065c801640 10 librados: watch_flush enter 2024-03-29T10:42:06.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:06.067+0000 7f065c801640 10 librados: watch_flush exit 2024-03-29T10:42:06.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:06.067+0000 7f065c801640 20 client.4328.objecter shutdown clearing up homeless session... 2024-03-29T10:42:06.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:06.067+0000 7f065c801640 10 client.4328.objecter successfully canceled tick 2024-03-29T10:42:06.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:06.067+0000 7f065c801640 1 -- 172.21.15.57:0/2859802760 >> v1:172.21.15.179:6816/2698295936 conn(0x7f063c06db60 legacy=0x7f063c070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:06.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:06.067+0000 7f065c801640 10 monclient: shutdown 2024-03-29T10:42:06.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:06.067+0000 7f065c801640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:06.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:06.067+0000 7f065c801640 1 -- 172.21.15.57:0/2859802760 >> v1:172.21.15.57:6790/0 conn(0x7f0650006a70 legacy=0x7f06500c2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:06.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:06.068+0000 7f065c801640 1 -- 172.21.15.57:0/2859802760 shutdown_connections 2024-03-29T10:42:06.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:06.068+0000 7f065c801640 1 -- 172.21.15.57:0/2859802760 >> 172.21.15.57:0/2859802760 conn(0x7f06500058f0 msgr2=0x7f06500c3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:06.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:06.068+0000 7f065c801640 1 -- 172.21.15.57:0/2859802760 shutdown_connections 2024-03-29T10:42:06.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:06.068+0000 7f065c801640 1 -- 172.21.15.57:0/2859802760 wait complete. 2024-03-29T10:42:06.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:06.068+0000 7f065c801640 1 librados: shutdown 2024-03-29T10:42:06.070 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:12.877 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:13.025 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.041+0000 7f15e0496640 10 monclient: get_monmap_and_config 2024-03-29T10:42:13.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.041+0000 7f15e0496640 10 monclient: build_initial_monmap 2024-03-29T10:42:13.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.041+0000 7f15e0496640 10 monclient: monmap: 2024-03-29T10:42:13.026 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:42:13.026 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:13.026 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:42:13.042981+0000 2024-03-29T10:42:13.026 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:42:13.042981+0000 2024-03-29T10:42:13.026 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:13.026 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:13.026 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:42:13.026 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:42:13.026 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:42:13.027 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:13.027 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.042+0000 7f15e0496640 1 Processor -- start 2024-03-29T10:42:13.028 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.042+0000 7f15e0496640 1 -- start start 2024-03-29T10:42:13.028 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.042+0000 7f15e0496640 10 monclient: init 2024-03-29T10:42:13.028 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.043+0000 7f15e0496640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:13.028 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.043+0000 7f15e0496640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:42:13.028 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.043+0000 7f15e0496640 10 monclient(hunting): picked mon.noname-a con 0x7f15d80a9ac0 addr v1:172.21.15.57:6789/0 2024-03-29T10:42:13.028 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.043+0000 7f15e0496640 10 monclient(hunting): picked mon.noname-c con 0x7f15d80b4100 addr v1:172.21.15.179:6789/0 2024-03-29T10:42:13.028 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.043+0000 7f15e0496640 10 monclient(hunting): picked mon.noname-b con 0x7f15d813ee90 addr v1:172.21.15.57:6790/0 2024-03-29T10:42:13.028 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.043+0000 7f15e0496640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15d80af6f0 con 0x7f15d80a9ac0 2024-03-29T10:42:13.029 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.043+0000 7f15e0496640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15d80aac80 con 0x7f15d80b4100 2024-03-29T10:42:13.029 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.043+0000 7f15e0496640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15d80aaf60 con 0x7f15d813ee90 2024-03-29T10:42:13.029 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.043+0000 7f15e0496640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:13.029 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.043+0000 7f15e0496640 10 monclient(hunting): authenticate will time out at 1085.220727s 2024-03-29T10:42:13.029 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.045+0000 7f15dda31640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f15d80b4100 0x7f15d8146ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:37256/0 (socket says 172.21.15.57:37256) 2024-03-29T10:42:13.030 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.045+0000 7f15dda31640 1 -- 172.21.15.57:0/1967875130 learned_addr learned my addr 172.21.15.57:0/1967875130 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:42:13.031 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.046+0000 7f15dd230640 1 -- 172.21.15.57:0/1967875130 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428686792 0 0) 0x7f15d80aac80 con 0x7f15d80b4100 2024-03-29T10:42:13.031 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.046+0000 7f15dd230640 10 monclient(hunting): _negotiate 2024-03-29T10:42:13.031 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.046+0000 7f15dd230640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:13.031 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.046+0000 7f15dd230640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:13.031 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.047+0000 7f15dd230640 10 monclient(hunting): my global_id is 4324 2024-03-29T10:42:13.031 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.047+0000 7f15dd230640 1 -- 172.21.15.57:0/1967875130 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15c4003520 con 0x7f15d80b4100 2024-03-29T10:42:13.032 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.047+0000 7f15dd230640 1 -- 172.21.15.57:0/1967875130 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3707267743 0 0) 0x7f15c4003520 con 0x7f15d80b4100 2024-03-29T10:42:13.032 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.047+0000 7f15dd230640 1 -- 172.21.15.57:0/1967875130 >> v1:172.21.15.57:6790/0 conn(0x7f15d813ee90 legacy=0x7f15d814a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:13.032 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.047+0000 7f15dd230640 1 -- 172.21.15.57:0/1967875130 >> v1:172.21.15.57:6789/0 conn(0x7f15d80a9ac0 legacy=0x7f15d80adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:13.032 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.048+0000 7f15dd230640 10 monclient: _finish_hunting 0 2024-03-29T10:42:13.032 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.048+0000 7f15dd230640 1 monclient: found mon.noname-c 2024-03-29T10:42:13.033 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.048+0000 7f15dd230640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:13.033 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.048+0000 7f15dd230640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:42:13.033 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.048+0000 7f15dd230640 1 -- 172.21.15.57:0/1967875130 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d813e970 con 0x7f15d80b4100 2024-03-29T10:42:13.033 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.048+0000 7f15dd230640 10 monclient: _finish_auth 0 2024-03-29T10:42:13.033 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.048+0000 7f15dd230640 10 monclient: _check_auth_tickets 2024-03-29T10:42:13.033 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.048+0000 7f15dd230640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:13.033 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.048+0000 7f15dd230640 1 -- 172.21.15.57:0/1967875130 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f15c8001690 con 0x7f15d80b4100 2024-03-29T10:42:13.033 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.048+0000 7f15dd230640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:13.033 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.048+0000 7f15dd230640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:13.033 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.048+0000 7f15dd230640 10 monclient: dump: 2024-03-29T10:42:13.033 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:13.033 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:13.033 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:13.033 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:13.034 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:13.034 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:13.034 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:13.034 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:13.034 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:13.034 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:13.034 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.048+0000 7f15e0496640 5 monclient: authenticate success, global_id 4324 2024-03-29T10:42:13.034 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.048+0000 7f15e0496640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:13.036 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.050+0000 7f15dd230640 1 -- 172.21.15.57:0/1967875130 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15c80018b0 con 0x7f15d80b4100 2024-03-29T10:42:13.036 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.050+0000 7f15dd230640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:13.036 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.050+0000 7f15dd230640 1 -- 172.21.15.57:0/1967875130 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f15c8001c40 con 0x7f15d80b4100 2024-03-29T10:42:13.036 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.050+0000 7f15dd230640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:13.036 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.050+0000 7f15dd230640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:13.036 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.050+0000 7f15dd230640 10 monclient: dump: 2024-03-29T10:42:13.037 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:13.037 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:13.037 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:13.037 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:13.037 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:13.037 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:13.037 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:13.037 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:13.037 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:13.037 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:13.037 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.050+0000 7f15e0496640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:13.037 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.050+0000 7f15e0496640 4 set_mon_vals no callback set 2024-03-29T10:42:13.037 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.051+0000 7f15e0496640 10 monclient: shutdown 2024-03-29T10:42:13.037 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.051+0000 7f15e0496640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:13.038 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.051+0000 7f15e0496640 1 -- 172.21.15.57:0/1967875130 >> v1:172.21.15.179:6789/0 conn(0x7f15d80b4100 legacy=0x7f15d8146ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:13.038 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.052+0000 7f15e0496640 1 -- 172.21.15.57:0/1967875130 shutdown_connections 2024-03-29T10:42:13.038 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.052+0000 7f15e0496640 1 -- 172.21.15.57:0/1967875130 >> 172.21.15.57:0/1967875130 conn(0x7f15d80b35e0 msgr2=0x7f15d80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:13.039 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.052+0000 7f15e0496640 1 -- 172.21.15.57:0/1967875130 shutdown_connections 2024-03-29T10:42:13.039 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.052+0000 7f15e0496640 1 -- 172.21.15.57:0/1967875130 wait complete. 2024-03-29T10:42:13.039 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.053+0000 7f15e0496640 10 monclient: build_initial_monmap 2024-03-29T10:42:13.039 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.053+0000 7f15e0496640 10 monclient: monmap: 2024-03-29T10:42:13.040 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:42:13.040 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:13.040 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:42:13.040 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:42:13.040 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:13.040 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:13.040 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:42:13.040 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:42:13.040 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:42:13.040 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:13.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.054+0000 7f15e0496640 1 librados: starting msgr at 2024-03-29T10:42:13.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.054+0000 7f15e0496640 1 librados: starting objecter 2024-03-29T10:42:13.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.054+0000 7f15e0496640 1 Processor -- start 2024-03-29T10:42:13.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.054+0000 7f15e0496640 1 -- start start 2024-03-29T10:42:13.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.054+0000 7f15e0496640 1 librados: setting wanted keys 2024-03-29T10:42:13.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.054+0000 7f15e0496640 1 librados: calling monclient init 2024-03-29T10:42:13.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.054+0000 7f15e0496640 10 monclient: init 2024-03-29T10:42:13.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.055+0000 7f15e0496640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:13.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.055+0000 7f15e0496640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:42:13.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.056+0000 7f15e0496640 10 monclient(hunting): picked mon.noname-a con 0x7f15d80a9ac0 addr v1:172.21.15.57:6789/0 2024-03-29T10:42:13.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.056+0000 7f15e0496640 10 monclient(hunting): picked mon.noname-b con 0x7f15d80b4100 addr v1:172.21.15.179:6789/0 2024-03-29T10:42:13.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.056+0000 7f15e0496640 10 monclient(hunting): picked mon.noname-c con 0x7f15d813ee90 addr v1:172.21.15.57:6790/0 2024-03-29T10:42:13.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.056+0000 7f15e0496640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15d8140f70 con 0x7f15d80a9ac0 2024-03-29T10:42:13.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.056+0000 7f15e0496640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15d81411e0 con 0x7f15d80b4100 2024-03-29T10:42:13.043 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.056+0000 7f15e0496640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15d81414c0 con 0x7f15d813ee90 2024-03-29T10:42:13.043 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.056+0000 7f15e0496640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:13.043 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.056+0000 7f15e0496640 10 monclient(hunting): authenticate will time out at 1085.233652s 2024-03-29T10:42:13.043 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.057+0000 7f15dda31640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f15d80b4100 0x7f15d813cf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:37644/0 (socket says 172.21.15.57:37644) 2024-03-29T10:42:13.043 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.057+0000 7f15dda31640 1 -- 172.21.15.57:0/3416012571 learned_addr learned my addr 172.21.15.57:0/3416012571 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:42:13.043 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.058+0000 7f15ceffd640 10 client.?.objecter ms_handle_connect 0x7f15d80b4100 2024-03-29T10:42:13.043 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.058+0000 7f15ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:13.043 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.058+0000 7f15ceffd640 1 -- 172.21.15.57:0/3416012571 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306753693 0 0) 0x7f15d81411e0 con 0x7f15d80b4100 2024-03-29T10:42:13.043 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.058+0000 7f15ceffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:13.043 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.058+0000 7f15ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:13.043 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.058+0000 7f15ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:13.043 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.058+0000 7f15ceffd640 10 monclient(hunting): my global_id is 4327 2024-03-29T10:42:13.043 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.058+0000 7f15ceffd640 1 -- 172.21.15.57:0/3416012571 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15bc004eb0 con 0x7f15d80b4100 2024-03-29T10:42:13.043 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.058+0000 7f15ceffd640 10 client.?.objecter ms_handle_connect 0x7f15d80a9ac0 2024-03-29T10:42:13.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.058+0000 7f15ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:13.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.059+0000 7f15ceffd640 1 -- 172.21.15.57:0/3416012571 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4134618382 0 0) 0x7f15bc004eb0 con 0x7f15d80b4100 2024-03-29T10:42:13.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.059+0000 7f15ceffd640 1 -- 172.21.15.57:0/3416012571 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15d81411e0 con 0x7f15d80b4100 2024-03-29T10:42:13.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.059+0000 7f15ceffd640 1 -- 172.21.15.57:0/3416012571 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f15c8001bc0 con 0x7f15d80b4100 2024-03-29T10:42:13.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.059+0000 7f15ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:13.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.059+0000 7f15ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:13.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.059+0000 7f15ceffd640 10 monclient(hunting): dump: 2024-03-29T10:42:13.047 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:13.047 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:13.047 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:13.047 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:13.047 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:13.047 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:13.047 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:13.047 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:13.047 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:13.047 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:13.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.059+0000 7f15ceffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:13.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.059+0000 7f15ceffd640 1 -- 172.21.15.57:0/3416012571 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3135610417 0 0) 0x7f15d81411e0 con 0x7f15d80b4100 2024-03-29T10:42:13.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.060+0000 7f15ceffd640 1 -- 172.21.15.57:0/3416012571 >> v1:172.21.15.57:6790/0 conn(0x7f15d813ee90 legacy=0x7f15d81e6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:13.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.060+0000 7f15ceffd640 1 -- 172.21.15.57:0/3416012571 >> v1:172.21.15.57:6789/0 conn(0x7f15d80a9ac0 legacy=0x7f15d812eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:13.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.060+0000 7f15ceffd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:13.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.060+0000 7f15ceffd640 1 monclient: found mon.b 2024-03-29T10:42:13.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.060+0000 7f15ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:13.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.060+0000 7f15ceffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:42:13.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.060+0000 7f15ceffd640 1 -- 172.21.15.57:0/3416012571 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d81417a0 con 0x7f15d80b4100 2024-03-29T10:42:13.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.060+0000 7f15ceffd640 10 monclient: _finish_auth 0 2024-03-29T10:42:13.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.060+0000 7f15ceffd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:13.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.060+0000 7f15ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:13.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.060+0000 7f15ceffd640 10 client.?.objecter ms_handle_connect 0x7f15d813ee90 2024-03-29T10:42:13.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.060+0000 7f15e0496640 5 monclient: authenticate success, global_id 4327 2024-03-29T10:42:13.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.060+0000 7f15e0496640 10 monclient: _renew_subs 2024-03-29T10:42:13.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.060+0000 7f15e0496640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:42:13.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.060+0000 7f15e0496640 1 -- 172.21.15.57:0/3416012571 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d8141a80 con 0x7f15d80b4100 2024-03-29T10:42:13.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.060+0000 7f15e0496640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:13.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.060+0000 7f15e0496640 10 monclient: _renew_subs 2024-03-29T10:42:13.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.060+0000 7f15e0496640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:42:13.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.060+0000 7f15e0496640 1 -- 172.21.15.57:0/3416012571 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d813d630 con 0x7f15d80b4100 2024-03-29T10:42:13.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.060+0000 7f15e0496640 1 librados: init done 2024-03-29T10:42:13.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.061+0000 7f15ceffd640 10 client.4327.objecter resend_mon_ops 2024-03-29T10:42:13.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.062+0000 7f15e0496640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:13.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.062+0000 7f15e0496640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:13.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.062+0000 7f15e0496640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:42:13.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.062+0000 7f15e0496640 1 -- 172.21.15.57:0/3416012571 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d8075f10 con 0x7f15d80b4100 2024-03-29T10:42:13.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.064+0000 7f15ceffd640 1 -- 172.21.15.57:0/3416012571 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15c8004bf0 con 0x7f15d80b4100 2024-03-29T10:42:13.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.065+0000 7f15ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:13.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.065+0000 7f15ceffd640 1 -- 172.21.15.57:0/3416012571 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f15c8004f00 con 0x7f15d80b4100 2024-03-29T10:42:13.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.065+0000 7f15ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:13.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.065+0000 7f15ceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:13.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.065+0000 7f15ceffd640 10 monclient: dump: 2024-03-29T10:42:13.051 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:13.051 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:13.052 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:13.052 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:13.052 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:13.052 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:13.052 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:13.052 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:13.052 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:13.052 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:13.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.065+0000 7f15ceffd640 1 -- 172.21.15.57:0/3416012571 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f15c801aed0 con 0x7f15d80b4100 2024-03-29T10:42:13.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.066+0000 7f15dca2f640 4 set_mon_vals no callback set 2024-03-29T10:42:13.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.066+0000 7f15de232640 10 monclient: get_auth_request con 0x7f15bc06dcc0 auth_method 0 2024-03-29T10:42:13.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.066+0000 7f15ceffd640 1 -- 172.21.15.57:0/3416012571 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f15c8084c90 con 0x7f15d80b4100 2024-03-29T10:42:13.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.066+0000 7f15ceffd640 10 client.4327.objecter ms_dispatch 0x7f15d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:13.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.067+0000 7f15ceffd640 3 client.4327.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:13.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.067+0000 7f15ceffd640 3 client.4327.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:13.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.067+0000 7f15ceffd640 20 client.4327.objecter dump_active .. 0 homeless 2024-03-29T10:42:13.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.067+0000 7f15ceffd640 10 client.4327.objecter ms_handle_connect 0x7f15bc06dcc0 2024-03-29T10:42:13.059 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.075+0000 7f15ceffd640 1 -- 172.21.15.57:0/3416012571 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f15c8050400 con 0x7f15d80b4100 2024-03-29T10:42:13.059 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.075+0000 7f15ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:13.059 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.075+0000 7f15ceffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:13.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.409+0000 7f15e0496640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:13.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.409+0000 7f15e0496640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:13.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.409+0000 7f15e0496640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:42:13.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.409+0000 7f15e0496640 1 -- 172.21.15.57:0/3416012571 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15d814fb20 con 0x7f15d80b4100 2024-03-29T10:42:13.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.412+0000 7f15ceffd640 1 -- 172.21.15.57:0/3416012571 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f15c80524a0 con 0x7f15d80b4100 2024-03-29T10:42:13.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.412+0000 7f15ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:13.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.412+0000 7f15ceffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:13.397 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:42:13.397 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:13.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.415+0000 7f15e0496640 10 librados: watch_flush enter 2024-03-29T10:42:13.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.418+0000 7f15e0496640 10 librados: watch_flush exit 2024-03-29T10:42:13.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.418+0000 7f15e0496640 20 client.4327.objecter shutdown clearing up homeless session... 2024-03-29T10:42:13.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.418+0000 7f15e0496640 10 client.4327.objecter successfully canceled tick 2024-03-29T10:42:13.403 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.419+0000 7f15e0496640 1 -- 172.21.15.57:0/3416012571 >> v1:172.21.15.179:6816/2698295936 conn(0x7f15bc06dcc0 legacy=0x7f15bc070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:13.403 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.419+0000 7f15e0496640 10 monclient: shutdown 2024-03-29T10:42:13.403 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.419+0000 7f15e0496640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:13.403 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.419+0000 7f15e0496640 1 -- 172.21.15.57:0/3416012571 >> v1:172.21.15.179:6789/0 conn(0x7f15d80b4100 legacy=0x7f15d813cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:13.403 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.419+0000 7f15e0496640 1 -- 172.21.15.57:0/3416012571 shutdown_connections 2024-03-29T10:42:13.404 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.419+0000 7f15e0496640 1 -- 172.21.15.57:0/3416012571 >> 172.21.15.57:0/3416012571 conn(0x7f15d80b35e0 msgr2=0x7f15d80b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:13.404 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.420+0000 7f15e0496640 1 -- 172.21.15.57:0/3416012571 shutdown_connections 2024-03-29T10:42:13.404 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.420+0000 7f15e0496640 1 -- 172.21.15.57:0/3416012571 wait complete. 2024-03-29T10:42:13.404 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:13.420+0000 7f15e0496640 1 librados: shutdown 2024-03-29T10:42:13.421 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:20.228 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:20.373 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.387+0000 7fd55d633640 10 monclient: get_monmap_and_config 2024-03-29T10:42:20.373 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.387+0000 7fd55d633640 10 monclient: build_initial_monmap 2024-03-29T10:42:20.373 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.387+0000 7fd55d633640 10 monclient: monmap: 2024-03-29T10:42:20.373 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:42:20.373 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:20.373 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:42:20.389028+0000 2024-03-29T10:42:20.373 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:42:20.389028+0000 2024-03-29T10:42:20.373 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:20.374 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:20.374 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:42:20.374 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:42:20.374 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:42:20.374 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:20.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.390+0000 7fd55d633640 1 Processor -- start 2024-03-29T10:42:20.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.391+0000 7fd55d633640 1 -- start start 2024-03-29T10:42:20.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.391+0000 7fd55d633640 10 monclient: init 2024-03-29T10:42:20.376 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.392+0000 7fd55d633640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:20.377 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.392+0000 7fd55d633640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:42:20.377 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.392+0000 7fd55d633640 10 monclient(hunting): picked mon.noname-a con 0x7fd5580a9ac0 addr v1:172.21.15.57:6789/0 2024-03-29T10:42:20.377 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.392+0000 7fd55d633640 10 monclient(hunting): picked mon.noname-b con 0x7fd55813ee90 addr v1:172.21.15.57:6790/0 2024-03-29T10:42:20.377 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.392+0000 7fd55d633640 10 monclient(hunting): picked mon.noname-c con 0x7fd5580af900 addr v1:172.21.15.179:6789/0 2024-03-29T10:42:20.377 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.392+0000 7fd55d633640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5580afd10 con 0x7fd5580a9ac0 2024-03-29T10:42:20.377 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.392+0000 7fd55d633640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5580af6f0 con 0x7fd5580af900 2024-03-29T10:42:20.377 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.392+0000 7fd55d633640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5580aac80 con 0x7fd55813ee90 2024-03-29T10:42:20.377 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.392+0000 7fd55d633640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:20.377 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.392+0000 7fd55d633640 10 monclient(hunting): authenticate will time out at 1092.569858s 2024-03-29T10:42:20.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.394+0000 7fd5577fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fd5580af900 0x7fd558149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:53906/0 (socket says 172.21.15.57:53906) 2024-03-29T10:42:20.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.394+0000 7fd5577fe640 1 -- 172.21.15.57:0/924682132 learned_addr learned my addr 172.21.15.57:0/924682132 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:42:20.379 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.395+0000 7fd555ffb640 1 -- 172.21.15.57:0/924682132 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193613795 0 0) 0x7fd5580af6f0 con 0x7fd5580af900 2024-03-29T10:42:20.379 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.395+0000 7fd555ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:42:20.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.395+0000 7fd555ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:20.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.395+0000 7fd555ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:20.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.395+0000 7fd555ffb640 10 monclient(hunting): my global_id is 4330 2024-03-29T10:42:20.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.395+0000 7fd555ffb640 1 -- 172.21.15.57:0/924682132 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd540003520 con 0x7fd5580af900 2024-03-29T10:42:20.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.395+0000 7fd555ffb640 1 -- 172.21.15.57:0/924682132 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2206856527 0 0) 0x7fd540003520 con 0x7fd5580af900 2024-03-29T10:42:20.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.395+0000 7fd555ffb640 1 -- 172.21.15.57:0/924682132 >> v1:172.21.15.57:6790/0 conn(0x7fd55813ee90 legacy=0x7fd55813b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:20.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.395+0000 7fd555ffb640 1 -- 172.21.15.57:0/924682132 >> v1:172.21.15.57:6789/0 conn(0x7fd5580a9ac0 legacy=0x7fd5580adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:20.382 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.396+0000 7fd555ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:42:20.382 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.396+0000 7fd555ffb640 1 monclient: found mon.noname-c 2024-03-29T10:42:20.382 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.396+0000 7fd555ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:20.383 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.396+0000 7fd555ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:42:20.383 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.396+0000 7fd555ffb640 1 -- 172.21.15.57:0/924682132 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5580aaf60 con 0x7fd5580af900 2024-03-29T10:42:20.383 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.396+0000 7fd555ffb640 10 monclient: _finish_auth 0 2024-03-29T10:42:20.383 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.396+0000 7fd555ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:42:20.383 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.396+0000 7fd555ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:20.383 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.396+0000 7fd555ffb640 1 -- 172.21.15.57:0/924682132 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd548001690 con 0x7fd5580af900 2024-03-29T10:42:20.383 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.396+0000 7fd555ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:20.383 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.396+0000 7fd555ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:20.383 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.396+0000 7fd555ffb640 10 monclient: dump: 2024-03-29T10:42:20.383 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:20.383 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:20.383 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:20.383 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:20.383 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:20.383 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:20.384 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:20.384 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:20.384 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:20.384 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:20.384 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.396+0000 7fd55d633640 5 monclient: authenticate success, global_id 4330 2024-03-29T10:42:20.384 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.396+0000 7fd55d633640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:20.384 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.398+0000 7fd555ffb640 1 -- 172.21.15.57:0/924682132 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd548001860 con 0x7fd5580af900 2024-03-29T10:42:20.384 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.398+0000 7fd555ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:20.384 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.398+0000 7fd55d633640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:20.384 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.398+0000 7fd55d633640 4 set_mon_vals no callback set 2024-03-29T10:42:20.384 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.398+0000 7fd555ffb640 1 -- 172.21.15.57:0/924682132 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd548001cb0 con 0x7fd5580af900 2024-03-29T10:42:20.385 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.398+0000 7fd555ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:20.385 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.398+0000 7fd555ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:20.385 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.398+0000 7fd555ffb640 10 monclient: dump: 2024-03-29T10:42:20.385 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:20.385 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:20.385 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:20.385 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:20.385 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:20.385 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:20.385 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:20.385 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:20.385 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:20.385 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:20.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.399+0000 7fd55d633640 10 monclient: shutdown 2024-03-29T10:42:20.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.399+0000 7fd55d633640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:20.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.399+0000 7fd55d633640 1 -- 172.21.15.57:0/924682132 >> v1:172.21.15.179:6789/0 conn(0x7fd5580af900 legacy=0x7fd558149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:20.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.399+0000 7fd55d633640 1 -- 172.21.15.57:0/924682132 shutdown_connections 2024-03-29T10:42:20.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.399+0000 7fd55d633640 1 -- 172.21.15.57:0/924682132 >> 172.21.15.57:0/924682132 conn(0x7fd5580b35e0 msgr2=0x7fd5580b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:20.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.400+0000 7fd55d633640 1 -- 172.21.15.57:0/924682132 shutdown_connections 2024-03-29T10:42:20.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.400+0000 7fd55d633640 1 -- 172.21.15.57:0/924682132 wait complete. 2024-03-29T10:42:20.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.401+0000 7fd55d633640 10 monclient: build_initial_monmap 2024-03-29T10:42:20.388 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.401+0000 7fd55d633640 10 monclient: monmap: 2024-03-29T10:42:20.388 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:42:20.388 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:20.388 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:42:20.388 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:42:20.388 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:20.388 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:20.388 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:42:20.388 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:42:20.388 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:42:20.388 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:20.388 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.402+0000 7fd55d633640 1 librados: starting msgr at 2024-03-29T10:42:20.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.402+0000 7fd55d633640 1 librados: starting objecter 2024-03-29T10:42:20.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.403+0000 7fd55d633640 1 Processor -- start 2024-03-29T10:42:20.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.403+0000 7fd55d633640 1 -- start start 2024-03-29T10:42:20.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.403+0000 7fd55d633640 1 librados: setting wanted keys 2024-03-29T10:42:20.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.403+0000 7fd55d633640 1 librados: calling monclient init 2024-03-29T10:42:20.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.403+0000 7fd55d633640 10 monclient: init 2024-03-29T10:42:20.390 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.405+0000 7fd55d633640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:20.390 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.405+0000 7fd55d633640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:42:20.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.405+0000 7fd55d633640 10 monclient(hunting): picked mon.noname-c con 0x7fd5580a9ac0 addr v1:172.21.15.57:6790/0 2024-03-29T10:42:20.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.406+0000 7fd55d633640 10 monclient(hunting): picked mon.noname-b con 0x7fd5580af900 addr v1:172.21.15.179:6789/0 2024-03-29T10:42:20.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.406+0000 7fd55d633640 10 monclient(hunting): picked mon.noname-a con 0x7fd55813ee90 addr v1:172.21.15.57:6789/0 2024-03-29T10:42:20.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.406+0000 7fd55d633640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd558141d90 con 0x7fd55813ee90 2024-03-29T10:42:20.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.406+0000 7fd55d633640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd558142000 con 0x7fd5580af900 2024-03-29T10:42:20.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.406+0000 7fd55d633640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55812b780 con 0x7fd5580a9ac0 2024-03-29T10:42:20.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.406+0000 7fd55d633640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:20.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.406+0000 7fd55d633640 10 monclient(hunting): authenticate will time out at 1092.583427s 2024-03-29T10:42:20.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.408+0000 7fd5567fc640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fd5580af900 0x7fd55813fc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:54970/0 (socket says 172.21.15.57:54970) 2024-03-29T10:42:20.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.408+0000 7fd5567fc640 1 -- 172.21.15.57:0/3124548026 learned_addr learned my addr 172.21.15.57:0/3124548026 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:42:20.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.408+0000 7fd53f7fe640 10 client.?.objecter ms_handle_connect 0x7fd5580af900 2024-03-29T10:42:20.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.408+0000 7fd53f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:20.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.408+0000 7fd53f7fe640 1 -- 172.21.15.57:0/3124548026 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 779837590 0 0) 0x7fd558142000 con 0x7fd5580af900 2024-03-29T10:42:20.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.408+0000 7fd53f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:42:20.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.408+0000 7fd53f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:20.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.408+0000 7fd53f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:20.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.408+0000 7fd53f7fe640 10 monclient(hunting): my global_id is 4333 2024-03-29T10:42:20.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.408+0000 7fd53f7fe640 1 -- 172.21.15.57:0/3124548026 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd544004eb0 con 0x7fd5580af900 2024-03-29T10:42:20.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.409+0000 7fd53f7fe640 1 -- 172.21.15.57:0/3124548026 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3116248367 0 0) 0x7fd544004eb0 con 0x7fd5580af900 2024-03-29T10:42:20.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.409+0000 7fd53f7fe640 1 -- 172.21.15.57:0/3124548026 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd558142000 con 0x7fd5580af900 2024-03-29T10:42:20.395 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.409+0000 7fd53f7fe640 1 -- 172.21.15.57:0/3124548026 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd550001ce0 con 0x7fd5580af900 2024-03-29T10:42:20.395 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.409+0000 7fd53f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:20.395 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.409+0000 7fd53f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:20.395 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.409+0000 7fd53f7fe640 10 monclient(hunting): dump: 2024-03-29T10:42:20.395 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:20.395 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:20.395 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:20.395 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:20.395 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:20.395 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:20.395 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:20.395 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:20.395 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:20.396 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:20.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.409+0000 7fd53f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:20.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.410+0000 7fd53f7fe640 10 client.?.objecter ms_handle_connect 0x7fd5580a9ac0 2024-03-29T10:42:20.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.410+0000 7fd53f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:20.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.410+0000 7fd53f7fe640 1 -- 172.21.15.57:0/3124548026 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053818109 0 0) 0x7fd55812b780 con 0x7fd5580a9ac0 2024-03-29T10:42:20.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.410+0000 7fd53f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:42:20.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.410+0000 7fd53f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:20.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.410+0000 7fd53f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:20.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.410+0000 7fd53f7fe640 10 monclient(hunting): my global_id is 4337 2024-03-29T10:42:20.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.410+0000 7fd53f7fe640 1 -- 172.21.15.57:0/3124548026 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd544004eb0 con 0x7fd5580a9ac0 2024-03-29T10:42:20.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.411+0000 7fd53f7fe640 1 -- 172.21.15.57:0/3124548026 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1027644485 0 0) 0x7fd544004eb0 con 0x7fd5580a9ac0 2024-03-29T10:42:20.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.411+0000 7fd53f7fe640 1 -- 172.21.15.57:0/3124548026 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd55812b780 con 0x7fd5580a9ac0 2024-03-29T10:42:20.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.411+0000 7fd53f7fe640 1 -- 172.21.15.57:0/3124548026 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd54c0016f0 con 0x7fd5580a9ac0 2024-03-29T10:42:20.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.411+0000 7fd53f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:20.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.411+0000 7fd53f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:20.398 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.411+0000 7fd53f7fe640 10 monclient(hunting): dump: 2024-03-29T10:42:20.398 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:20.398 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:20.398 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:20.398 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:20.398 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:20.398 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:20.398 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:20.398 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:20.398 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:20.398 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:20.398 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.411+0000 7fd53f7fe640 1 -- 172.21.15.57:0/3124548026 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 60156977 0 0) 0x7fd558142000 con 0x7fd5580af900 2024-03-29T10:42:20.398 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.411+0000 7fd53f7fe640 1 -- 172.21.15.57:0/3124548026 >> v1:172.21.15.57:6790/0 conn(0x7fd5580a9ac0 legacy=0x7fd558150d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:20.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.411+0000 7fd53f7fe640 1 -- 172.21.15.57:0/3124548026 >> v1:172.21.15.57:6789/0 conn(0x7fd55813ee90 legacy=0x7fd558140390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:20.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.412+0000 7fd53f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:42:20.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.412+0000 7fd53f7fe640 1 monclient: found mon.b 2024-03-29T10:42:20.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.412+0000 7fd53f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:20.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.412+0000 7fd53f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:42:20.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.412+0000 7fd53f7fe640 1 -- 172.21.15.57:0/3124548026 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd55812b910 con 0x7fd5580af900 2024-03-29T10:42:20.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.412+0000 7fd53f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:42:20.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.412+0000 7fd53f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:42:20.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.412+0000 7fd53f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:20.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.412+0000 7fd55d633640 5 monclient: authenticate success, global_id 4333 2024-03-29T10:42:20.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.412+0000 7fd55d633640 10 monclient: _renew_subs 2024-03-29T10:42:20.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.412+0000 7fd55d633640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:42:20.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.412+0000 7fd55d633640 1 -- 172.21.15.57:0/3124548026 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5581e3390 con 0x7fd5580af900 2024-03-29T10:42:20.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.412+0000 7fd55d633640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:20.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.412+0000 7fd55d633640 10 monclient: _renew_subs 2024-03-29T10:42:20.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.412+0000 7fd55d633640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:42:20.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.412+0000 7fd55d633640 1 -- 172.21.15.57:0/3124548026 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5581e3700 con 0x7fd5580af900 2024-03-29T10:42:20.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.412+0000 7fd55d633640 1 librados: init done 2024-03-29T10:42:20.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.412+0000 7fd53f7fe640 1 -- 172.21.15.57:0/3124548026 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd550001e40 con 0x7fd5580af900 2024-03-29T10:42:20.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.413+0000 7fd53f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:20.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.413+0000 7fd53f7fe640 1 -- 172.21.15.57:0/3124548026 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd550003dc0 con 0x7fd5580af900 2024-03-29T10:42:20.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.413+0000 7fd53f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:20.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.413+0000 7fd53f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:20.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.413+0000 7fd53f7fe640 10 monclient: dump: 2024-03-29T10:42:20.400 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:20.400 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:20.400 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:20.400 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:20.401 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:20.401 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:20.401 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:20.401 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:20.401 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:20.401 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:20.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.414+0000 7fd53f7fe640 1 -- 172.21.15.57:0/3124548026 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fd550003fa0 con 0x7fd5580af900 2024-03-29T10:42:20.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.415+0000 7fd5557fa640 4 set_mon_vals no callback set 2024-03-29T10:42:20.403 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.415+0000 7fd55d633640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:20.403 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.415+0000 7fd53f7fe640 1 -- 172.21.15.57:0/3124548026 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7fd550084e80 con 0x7fd5580af900 2024-03-29T10:42:20.403 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.415+0000 7fd53f7fe640 10 client.4333.objecter ms_dispatch 0x7fd558000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:20.403 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.415+0000 7fd53f7fe640 3 client.4333.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:20.403 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.416+0000 7fd53f7fe640 3 client.4333.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:20.403 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.416+0000 7fd53f7fe640 20 client.4333.objecter dump_active .. 0 homeless 2024-03-29T10:42:20.403 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.416+0000 7fd55d633640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:20.403 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.416+0000 7fd55d633640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:42:20.403 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.416+0000 7fd55d633640 1 -- 172.21.15.57:0/3124548026 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd538005080 con 0x7fd5580af900 2024-03-29T10:42:20.403 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.418+0000 7fd556ffd640 10 monclient: get_auth_request con 0x7fd54406de50 auth_method 0 2024-03-29T10:42:20.405 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.420+0000 7fd53f7fe640 10 client.4333.objecter ms_handle_connect 0x7fd54406de50 2024-03-29T10:42:20.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.426+0000 7fd53f7fe640 1 -- 172.21.15.57:0/3124548026 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd550050760 con 0x7fd5580af900 2024-03-29T10:42:20.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.426+0000 7fd53f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:20.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.426+0000 7fd53f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:20.738 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.754+0000 7fd55d633640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:20.738 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.754+0000 7fd55d633640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:20.738 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.754+0000 7fd55d633640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:42:20.738 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.754+0000 7fd55d633640 1 -- 172.21.15.57:0/3124548026 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd538005330 con 0x7fd5580af900 2024-03-29T10:42:20.739 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.755+0000 7fd53f7fe640 1 -- 172.21.15.57:0/3124548026 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7fd550052690 con 0x7fd5580af900 2024-03-29T10:42:20.739 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.755+0000 7fd53f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:20.739 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.755+0000 7fd53f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:20.739 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:42:20.739 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:20.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.758+0000 7fd55d633640 10 librados: watch_flush enter 2024-03-29T10:42:20.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.758+0000 7fd55d633640 10 librados: watch_flush exit 2024-03-29T10:42:20.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.758+0000 7fd55d633640 20 client.4333.objecter shutdown clearing up homeless session... 2024-03-29T10:42:20.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.758+0000 7fd55d633640 10 client.4333.objecter successfully canceled tick 2024-03-29T10:42:20.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.758+0000 7fd55d633640 1 -- 172.21.15.57:0/3124548026 >> v1:172.21.15.179:6816/2698295936 conn(0x7fd54406de50 legacy=0x7fd544070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:20.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.758+0000 7fd55d633640 10 monclient: shutdown 2024-03-29T10:42:20.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.758+0000 7fd55d633640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:20.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.758+0000 7fd55d633640 1 -- 172.21.15.57:0/3124548026 >> v1:172.21.15.179:6789/0 conn(0x7fd5580af900 legacy=0x7fd55813fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:20.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.758+0000 7fd55d633640 1 -- 172.21.15.57:0/3124548026 shutdown_connections 2024-03-29T10:42:20.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.758+0000 7fd55d633640 1 -- 172.21.15.57:0/3124548026 >> 172.21.15.57:0/3124548026 conn(0x7fd5580b35e0 msgr2=0x7fd5580b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:20.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.758+0000 7fd55d633640 1 -- 172.21.15.57:0/3124548026 shutdown_connections 2024-03-29T10:42:20.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.759+0000 7fd55d633640 1 -- 172.21.15.57:0/3124548026 wait complete. 2024-03-29T10:42:20.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:20.759+0000 7fd55d633640 1 librados: shutdown 2024-03-29T10:42:20.761 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:27.568 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:27.725 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.742+0000 7fb1b2ec1640 10 monclient: get_monmap_and_config 2024-03-29T10:42:27.725 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.742+0000 7fb1b2ec1640 10 monclient: build_initial_monmap 2024-03-29T10:42:27.725 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.742+0000 7fb1b2ec1640 10 monclient: monmap: 2024-03-29T10:42:27.726 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:42:27.726 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:27.726 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:42:27.743347+0000 2024-03-29T10:42:27.726 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:42:27.743347+0000 2024-03-29T10:42:27.726 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:27.726 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:27.726 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:42:27.726 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:42:27.726 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:42:27.726 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:27.728 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.745+0000 7fb1b2ec1640 1 Processor -- start 2024-03-29T10:42:27.728 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.745+0000 7fb1b2ec1640 1 -- start start 2024-03-29T10:42:27.728 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.745+0000 7fb1b2ec1640 10 monclient: init 2024-03-29T10:42:27.730 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.746+0000 7fb1b2ec1640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:27.730 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.746+0000 7fb1b2ec1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:42:27.730 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.746+0000 7fb1b2ec1640 10 monclient(hunting): picked mon.noname-c con 0x7fb1ac0b0520 addr v1:172.21.15.179:6789/0 2024-03-29T10:42:27.730 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.746+0000 7fb1b2ec1640 10 monclient(hunting): picked mon.noname-a con 0x7fb1ac0aed80 addr v1:172.21.15.57:6789/0 2024-03-29T10:42:27.730 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.746+0000 7fb1b2ec1640 10 monclient(hunting): picked mon.noname-b con 0x7fb1ac0729f0 addr v1:172.21.15.57:6790/0 2024-03-29T10:42:27.730 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.746+0000 7fb1b2ec1640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1ac144030 con 0x7fb1ac0aed80 2024-03-29T10:42:27.730 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.746+0000 7fb1b2ec1640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1ac06ec30 con 0x7fb1ac0b0520 2024-03-29T10:42:27.730 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.746+0000 7fb1b2ec1640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1ac06f860 con 0x7fb1ac0729f0 2024-03-29T10:42:27.730 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.746+0000 7fb1b2ec1640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:27.730 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.746+0000 7fb1b2ec1640 10 monclient(hunting): authenticate will time out at 1099.922578s 2024-03-29T10:42:27.733 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.750+0000 7fb1b26c0640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7fb1ac0729f0 0x7fb1ac151c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:51773/0 (socket says 172.21.15.57:51773) 2024-03-29T10:42:27.733 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.750+0000 7fb1b26c0640 1 -- 172.21.15.57:0/1989844144 learned_addr learned my addr 172.21.15.57:0/1989844144 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:42:27.734 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.750+0000 7fb1b0ebd640 1 -- 172.21.15.57:0/1989844144 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018100615 0 0) 0x7fb1ac06f860 con 0x7fb1ac0729f0 2024-03-29T10:42:27.734 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.750+0000 7fb1b0ebd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:27.734 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.750+0000 7fb1b0ebd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:27.734 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.750+0000 7fb1b0ebd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:27.734 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.750+0000 7fb1b0ebd640 10 monclient(hunting): my global_id is 4340 2024-03-29T10:42:27.734 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.750+0000 7fb1b0ebd640 1 -- 172.21.15.57:0/1989844144 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb198003520 con 0x7fb1ac0729f0 2024-03-29T10:42:27.735 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.750+0000 7fb1b0ebd640 1 -- 172.21.15.57:0/1989844144 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2330975010 0 0) 0x7fb198003520 con 0x7fb1ac0729f0 2024-03-29T10:42:27.735 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.750+0000 7fb1b0ebd640 1 -- 172.21.15.57:0/1989844144 >> v1:172.21.15.179:6789/0 conn(0x7fb1ac0b0520 legacy=0x7fb1ac06f150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:27.738 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.754+0000 7fb1b0ebd640 1 -- 172.21.15.57:0/1989844144 >> v1:172.21.15.57:6789/0 conn(0x7fb1ac0aed80 legacy=0x7fb1ac0722e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:27.738 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.754+0000 7fb1b0ebd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:27.738 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.754+0000 7fb1b0ebd640 1 monclient: found mon.noname-b 2024-03-29T10:42:27.738 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.754+0000 7fb1b0ebd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:27.738 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.754+0000 7fb1b0ebd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:42:27.738 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.754+0000 7fb1b0ebd640 1 -- 172.21.15.57:0/1989844144 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ac1498e0 con 0x7fb1ac0729f0 2024-03-29T10:42:27.738 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.754+0000 7fb1b0ebd640 10 monclient: _finish_auth 0 2024-03-29T10:42:27.738 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.754+0000 7fb1b0ebd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:27.738 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.754+0000 7fb1b0ebd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:27.739 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.754+0000 7fb1b0ebd640 1 -- 172.21.15.57:0/1989844144 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb1a4001a10 con 0x7fb1ac0729f0 2024-03-29T10:42:27.739 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.754+0000 7fb1b0ebd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:27.739 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.754+0000 7fb1b0ebd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:27.739 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.754+0000 7fb1b0ebd640 10 monclient: dump: 2024-03-29T10:42:27.739 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:27.739 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:27.739 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:27.739 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:27.739 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:27.739 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:27.739 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:27.739 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:27.739 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:27.739 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:27.739 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.754+0000 7fb1b2ec1640 5 monclient: authenticate success, global_id 4340 2024-03-29T10:42:27.740 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.754+0000 7fb1b2ec1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:27.740 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.754+0000 7fb1b0ebd640 1 -- 172.21.15.57:0/1989844144 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1a4001be0 con 0x7fb1ac0729f0 2024-03-29T10:42:27.740 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.754+0000 7fb1b0ebd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:27.740 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.754+0000 7fb1b0ebd640 1 -- 172.21.15.57:0/1989844144 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb1a4002030 con 0x7fb1ac0729f0 2024-03-29T10:42:27.740 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.754+0000 7fb1b0ebd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:27.740 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.754+0000 7fb1b0ebd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:27.740 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.754+0000 7fb1b0ebd640 10 monclient: dump: 2024-03-29T10:42:27.740 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:27.740 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:27.740 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:27.740 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:27.740 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:27.740 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:27.740 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:27.741 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:27.741 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:27.741 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:27.741 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.755+0000 7fb1b2ec1640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:27.741 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.755+0000 7fb1b2ec1640 4 set_mon_vals no callback set 2024-03-29T10:42:27.741 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.756+0000 7fb1b2ec1640 10 monclient: shutdown 2024-03-29T10:42:27.742 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.756+0000 7fb1b2ec1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:27.742 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.756+0000 7fb1b2ec1640 1 -- 172.21.15.57:0/1989844144 >> v1:172.21.15.57:6790/0 conn(0x7fb1ac0729f0 legacy=0x7fb1ac151c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:27.742 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.757+0000 7fb1b2ec1640 1 -- 172.21.15.57:0/1989844144 shutdown_connections 2024-03-29T10:42:27.742 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.757+0000 7fb1b2ec1640 1 -- 172.21.15.57:0/1989844144 >> 172.21.15.57:0/1989844144 conn(0x7fb1ac0b13a0 msgr2=0x7fb1ac0b17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:27.742 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.759+0000 7fb1b2ec1640 1 -- 172.21.15.57:0/1989844144 shutdown_connections 2024-03-29T10:42:27.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.759+0000 7fb1b2ec1640 1 -- 172.21.15.57:0/1989844144 wait complete. 2024-03-29T10:42:27.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.760+0000 7fb1b2ec1640 10 monclient: build_initial_monmap 2024-03-29T10:42:27.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.760+0000 7fb1b2ec1640 10 monclient: monmap: 2024-03-29T10:42:27.743 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:42:27.743 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:27.743 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:42:27.743 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:42:27.744 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:27.744 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:27.744 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:42:27.744 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:42:27.744 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:42:27.744 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:27.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.761+0000 7fb1b2ec1640 1 librados: starting msgr at 2024-03-29T10:42:27.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.761+0000 7fb1b2ec1640 1 librados: starting objecter 2024-03-29T10:42:27.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.761+0000 7fb1b2ec1640 1 Processor -- start 2024-03-29T10:42:27.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.761+0000 7fb1b2ec1640 1 -- start start 2024-03-29T10:42:27.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.761+0000 7fb1b2ec1640 1 librados: setting wanted keys 2024-03-29T10:42:27.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.761+0000 7fb1b2ec1640 1 librados: calling monclient init 2024-03-29T10:42:27.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.761+0000 7fb1b2ec1640 10 monclient: init 2024-03-29T10:42:27.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.762+0000 7fb1b2ec1640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:27.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.762+0000 7fb1b2ec1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:42:27.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.762+0000 7fb1b2ec1640 10 monclient(hunting): picked mon.noname-a con 0x7fb1ac0729f0 addr v1:172.21.15.57:6789/0 2024-03-29T10:42:27.754 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.762+0000 7fb1b2ec1640 10 monclient(hunting): picked mon.noname-c con 0x7fb1ac0aed80 addr v1:172.21.15.57:6790/0 2024-03-29T10:42:27.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.762+0000 7fb1b2ec1640 10 monclient(hunting): picked mon.noname-b con 0x7fb1ac0b0520 addr v1:172.21.15.179:6789/0 2024-03-29T10:42:27.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.762+0000 7fb1b2ec1640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1ac1db480 con 0x7fb1ac0729f0 2024-03-29T10:42:27.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.762+0000 7fb1b2ec1640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1ac145aa0 con 0x7fb1ac0b0520 2024-03-29T10:42:27.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.762+0000 7fb1b2ec1640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1ac1441e0 con 0x7fb1ac0aed80 2024-03-29T10:42:27.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.762+0000 7fb1b2ec1640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:27.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.762+0000 7fb1b2ec1640 10 monclient(hunting): authenticate will time out at 1099.938434s 2024-03-29T10:42:27.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.765+0000 7fb1b1ebf640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fb1ac0729f0 0x7fb1ac145390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:47339/0 (socket says 172.21.15.57:47339) 2024-03-29T10:42:27.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.765+0000 7fb1b1ebf640 1 -- 172.21.15.57:0/2192411834 learned_addr learned my addr 172.21.15.57:0/2192411834 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:42:27.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.765+0000 7fb1a27fc640 10 client.?.objecter ms_handle_connect 0x7fb1ac0729f0 2024-03-29T10:42:27.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.765+0000 7fb1a27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:27.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.765+0000 7fb1a27fc640 1 -- 172.21.15.57:0/2192411834 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1737401246 0 0) 0x7fb1ac1db480 con 0x7fb1ac0729f0 2024-03-29T10:42:27.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.765+0000 7fb1a27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:27.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.765+0000 7fb1a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:27.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.765+0000 7fb1a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:27.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.765+0000 7fb1a27fc640 10 monclient(hunting): my global_id is 4323 2024-03-29T10:42:27.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.765+0000 7fb1a27fc640 1 -- 172.21.15.57:0/2192411834 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb190004eb0 con 0x7fb1ac0729f0 2024-03-29T10:42:27.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.767+0000 7fb1a27fc640 10 client.?.objecter ms_handle_connect 0x7fb1ac0aed80 2024-03-29T10:42:27.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.767+0000 7fb1a27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:27.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.767+0000 7fb1a27fc640 1 -- 172.21.15.57:0/2192411834 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934949740 0 0) 0x7fb1ac1441e0 con 0x7fb1ac0aed80 2024-03-29T10:42:27.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.767+0000 7fb1a27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:27.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.767+0000 7fb1a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:27.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.767+0000 7fb1a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:27.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.767+0000 7fb1a27fc640 10 monclient(hunting): my global_id is 4343 2024-03-29T10:42:27.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.767+0000 7fb1a27fc640 1 -- 172.21.15.57:0/2192411834 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1900051d0 con 0x7fb1ac0aed80 2024-03-29T10:42:27.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.767+0000 7fb1a27fc640 10 client.?.objecter ms_handle_connect 0x7fb1ac0b0520 2024-03-29T10:42:27.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.767+0000 7fb1a27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:27.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.767+0000 7fb1a27fc640 1 -- 172.21.15.57:0/2192411834 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1348312370 0 0) 0x7fb1900051d0 con 0x7fb1ac0aed80 2024-03-29T10:42:27.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.767+0000 7fb1a27fc640 1 -- 172.21.15.57:0/2192411834 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1ac1441e0 con 0x7fb1ac0aed80 2024-03-29T10:42:27.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.767+0000 7fb1a27fc640 1 -- 172.21.15.57:0/2192411834 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb1a80018a0 con 0x7fb1ac0aed80 2024-03-29T10:42:27.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.767+0000 7fb1a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:27.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.767+0000 7fb1a27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:27.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.767+0000 7fb1a27fc640 10 monclient(hunting): dump: 2024-03-29T10:42:27.757 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:27.757 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:27.757 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:27.758 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:27.758 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:27.758 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:27.758 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:27.758 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:27.758 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:27.758 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:27.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1a27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:27.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1a27fc640 1 -- 172.21.15.57:0/2192411834 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1563813719 0 0) 0x7fb190004eb0 con 0x7fb1ac0729f0 2024-03-29T10:42:27.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1a27fc640 1 -- 172.21.15.57:0/2192411834 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1900051d0 con 0x7fb1ac0729f0 2024-03-29T10:42:27.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1a27fc640 1 -- 172.21.15.57:0/2192411834 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb19c001790 con 0x7fb1ac0729f0 2024-03-29T10:42:27.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:27.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1a27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:27.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1a27fc640 10 monclient(hunting): dump: 2024-03-29T10:42:27.759 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:27.759 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:27.759 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:27.759 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:27.759 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:27.759 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:27.759 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:27.759 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:27.759 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:27.759 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:27.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1a27fc640 1 -- 172.21.15.57:0/2192411834 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2373564823 0 0) 0x7fb1ac145aa0 con 0x7fb1ac0b0520 2024-03-29T10:42:27.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1a27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:27.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:27.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:27.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1a27fc640 10 monclient(hunting): my global_id is 4336 2024-03-29T10:42:27.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1a27fc640 1 -- 172.21.15.57:0/2192411834 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb190005d50 con 0x7fb1ac0b0520 2024-03-29T10:42:27.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1a27fc640 1 -- 172.21.15.57:0/2192411834 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3506076105 0 0) 0x7fb1ac1441e0 con 0x7fb1ac0aed80 2024-03-29T10:42:27.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1a27fc640 1 -- 172.21.15.57:0/2192411834 >> v1:172.21.15.179:6789/0 conn(0x7fb1ac0b0520 legacy=0x7fb1ac13f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:27.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1a27fc640 1 -- 172.21.15.57:0/2192411834 >> v1:172.21.15.57:6789/0 conn(0x7fb1ac0729f0 legacy=0x7fb1ac145390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:27.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1a27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:42:27.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1a27fc640 1 monclient: found mon.c 2024-03-29T10:42:27.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:27.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1a27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:42:27.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1a27fc640 1 -- 172.21.15.57:0/2192411834 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ac1dc020 con 0x7fb1ac0aed80 2024-03-29T10:42:27.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1a27fc640 10 monclient: _finish_auth 0 2024-03-29T10:42:27.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1a27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:42:27.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:27.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1b2ec1640 5 monclient: authenticate success, global_id 4343 2024-03-29T10:42:27.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1b2ec1640 10 monclient: _renew_subs 2024-03-29T10:42:27.761 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1b2ec1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:42:27.761 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1b2ec1640 1 -- 172.21.15.57:0/2192411834 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1ac145ce0 con 0x7fb1ac0aed80 2024-03-29T10:42:27.761 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1b2ec1640 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:27.761 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1b2ec1640 10 monclient: _renew_subs 2024-03-29T10:42:27.761 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1b2ec1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:42:27.761 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1b2ec1640 1 -- 172.21.15.57:0/2192411834 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1ac0ac480 con 0x7fb1ac0aed80 2024-03-29T10:42:27.761 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.768+0000 7fb1b2ec1640 1 librados: init done 2024-03-29T10:42:27.761 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.770+0000 7fb1a27fc640 1 -- 172.21.15.57:0/2192411834 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1a80035f0 con 0x7fb1ac0aed80 2024-03-29T10:42:27.761 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.770+0000 7fb1a27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:27.761 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.770+0000 7fb1a27fc640 1 -- 172.21.15.57:0/2192411834 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb1a80039c0 con 0x7fb1ac0aed80 2024-03-29T10:42:27.761 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.770+0000 7fb1a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:27.761 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.770+0000 7fb1a27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:27.761 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.770+0000 7fb1a27fc640 10 monclient: dump: 2024-03-29T10:42:27.761 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:27.762 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:27.762 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:27.762 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:27.762 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:27.762 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:27.762 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:27.762 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:27.762 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:27.762 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:27.762 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.770+0000 7fb1a3fff640 4 set_mon_vals no callback set 2024-03-29T10:42:27.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.771+0000 7fb1b2ec1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:27.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.771+0000 7fb1b2ec1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:27.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.771+0000 7fb1b2ec1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:42:27.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.771+0000 7fb1a27fc640 1 -- 172.21.15.57:0/2192411834 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fb1a8003be0 con 0x7fb1ac0aed80 2024-03-29T10:42:27.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.771+0000 7fb1b2ec1640 1 -- 172.21.15.57:0/2192411834 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb194005080 con 0x7fb1ac0aed80 2024-03-29T10:42:27.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.772+0000 7fb1a27fc640 1 -- 172.21.15.57:0/2192411834 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7fb1a8083750 con 0x7fb1ac0aed80 2024-03-29T10:42:27.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.772+0000 7fb1a27fc640 10 client.4343.objecter ms_dispatch 0x7fb1ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:27.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.772+0000 7fb1a27fc640 3 client.4343.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:27.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.772+0000 7fb1a27fc640 3 client.4343.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:27.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.772+0000 7fb1a27fc640 20 client.4343.objecter dump_active .. 0 homeless 2024-03-29T10:42:27.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.772+0000 7fb1b1ebf640 10 monclient: get_auth_request con 0x7fb19006e490 auth_method 0 2024-03-29T10:42:27.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.773+0000 7fb1a27fc640 10 client.4343.objecter ms_handle_connect 0x7fb19006e490 2024-03-29T10:42:27.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.781+0000 7fb1a27fc640 1 -- 172.21.15.57:0/2192411834 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb1a804f090 con 0x7fb1ac0aed80 2024-03-29T10:42:27.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.781+0000 7fb1a27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:27.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:27.781+0000 7fb1a27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:28.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:28.125+0000 7fb1b2ec1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:28.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:28.125+0000 7fb1b2ec1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:28.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:28.125+0000 7fb1b2ec1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:42:28.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:28.125+0000 7fb1b2ec1640 1 -- 172.21.15.57:0/2192411834 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb194005330 con 0x7fb1ac0aed80 2024-03-29T10:42:28.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:28.128+0000 7fb1a27fc640 1 -- 172.21.15.57:0/2192411834 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7fb1a8082e00 con 0x7fb1ac0aed80 2024-03-29T10:42:28.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:28.128+0000 7fb1a27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:28.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:28.128+0000 7fb1a27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:28.112 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:42:28.112 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:28.114 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:28.132+0000 7fb1b2ec1640 10 librados: watch_flush enter 2024-03-29T10:42:28.115 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:28.132+0000 7fb1b2ec1640 10 librados: watch_flush exit 2024-03-29T10:42:28.115 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:28.132+0000 7fb1b2ec1640 20 client.4343.objecter shutdown clearing up homeless session... 2024-03-29T10:42:28.115 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:28.132+0000 7fb1b2ec1640 10 client.4343.objecter successfully canceled tick 2024-03-29T10:42:28.116 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:28.132+0000 7fb1b2ec1640 1 -- 172.21.15.57:0/2192411834 >> v1:172.21.15.179:6816/2698295936 conn(0x7fb19006e490 legacy=0x7fb190070950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:28.116 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:28.132+0000 7fb1b2ec1640 10 monclient: shutdown 2024-03-29T10:42:28.116 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:28.132+0000 7fb1b2ec1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:28.116 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:28.132+0000 7fb1b2ec1640 1 -- 172.21.15.57:0/2192411834 >> v1:172.21.15.57:6790/0 conn(0x7fb1ac0aed80 legacy=0x7fb1ac1db910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:28.116 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:28.132+0000 7fb1b2ec1640 1 -- 172.21.15.57:0/2192411834 shutdown_connections 2024-03-29T10:42:28.116 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:28.132+0000 7fb1b2ec1640 1 -- 172.21.15.57:0/2192411834 >> 172.21.15.57:0/2192411834 conn(0x7fb1ac0b13a0 msgr2=0x7fb1ac071410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:28.116 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:28.132+0000 7fb1b2ec1640 1 -- 172.21.15.57:0/2192411834 shutdown_connections 2024-03-29T10:42:28.116 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:28.132+0000 7fb1b2ec1640 1 -- 172.21.15.57:0/2192411834 wait complete. 2024-03-29T10:42:28.116 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:28.132+0000 7fb1b2ec1640 1 librados: shutdown 2024-03-29T10:42:28.133 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:34.940 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:35.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.112+0000 7f1de66dc640 10 monclient: get_monmap_and_config 2024-03-29T10:42:35.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.112+0000 7f1de66dc640 10 monclient: build_initial_monmap 2024-03-29T10:42:35.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.112+0000 7f1de66dc640 10 monclient: monmap: 2024-03-29T10:42:35.097 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:42:35.097 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:35.097 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:42:35.112769+0000 2024-03-29T10:42:35.097 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:42:35.112769+0000 2024-03-29T10:42:35.097 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:35.097 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:35.097 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:42:35.097 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:42:35.097 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:42:35.098 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:35.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.113+0000 7f1de66dc640 1 Processor -- start 2024-03-29T10:42:35.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.114+0000 7f1de66dc640 1 -- start start 2024-03-29T10:42:35.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.114+0000 7f1de66dc640 10 monclient: init 2024-03-29T10:42:35.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.114+0000 7f1de66dc640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:35.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.114+0000 7f1de66dc640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:42:35.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.114+0000 7f1de66dc640 10 monclient(hunting): picked mon.noname-a con 0x7f1dd8029eb0 addr v1:172.21.15.57:6789/0 2024-03-29T10:42:35.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.114+0000 7f1de66dc640 10 monclient(hunting): picked mon.noname-c con 0x7f1dd802b0a0 addr v1:172.21.15.179:6789/0 2024-03-29T10:42:35.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.114+0000 7f1de66dc640 10 monclient(hunting): picked mon.noname-b con 0x7f1dd80c9570 addr v1:172.21.15.57:6790/0 2024-03-29T10:42:35.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.114+0000 7f1de66dc640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dd802a290 con 0x7f1dd8029eb0 2024-03-29T10:42:35.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.114+0000 7f1de66dc640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dd80331c0 con 0x7f1dd802b0a0 2024-03-29T10:42:35.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.114+0000 7f1de66dc640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dd802c100 con 0x7f1dd80c9570 2024-03-29T10:42:35.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.114+0000 7f1de66dc640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:35.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.114+0000 7f1de66dc640 10 monclient(hunting): authenticate will time out at 1107.290944s 2024-03-29T10:42:35.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.117+0000 7f1de56da640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1dd8029eb0 0x7f1dd80cf120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:54883/0 (socket says 172.21.15.57:54883) 2024-03-29T10:42:35.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1de56da640 1 -- 172.21.15.57:0/2192467053 learned_addr learned my addr 172.21.15.57:0/2192467053 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:42:35.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 1 -- 172.21.15.57:0/2192467053 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636299357 0 0) 0x7f1dd802a290 con 0x7f1dd8029eb0 2024-03-29T10:42:35.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:35.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:35.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:35.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 10 monclient(hunting): my global_id is 4326 2024-03-29T10:42:35.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 1 -- 172.21.15.57:0/2192467053 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dcc003520 con 0x7f1dd8029eb0 2024-03-29T10:42:35.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 1 -- 172.21.15.57:0/2192467053 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1402342223 0 0) 0x7f1dcc003520 con 0x7f1dd8029eb0 2024-03-29T10:42:35.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 1 -- 172.21.15.57:0/2192467053 >> v1:172.21.15.57:6790/0 conn(0x7f1dd80c9570 legacy=0x7f1dd80d74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:42:35.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 1 -- 172.21.15.57:0/2192467053 >> v1:172.21.15.179:6789/0 conn(0x7f1dd802b0a0 legacy=0x7f1dd80c8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:35.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:42:35.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 1 monclient: found mon.noname-a 2024-03-29T10:42:35.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:35.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:42:35.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 1 -- 172.21.15.57:0/2192467053 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd802a440 con 0x7f1dd8029eb0 2024-03-29T10:42:35.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 10 monclient: _finish_auth 0 2024-03-29T10:42:35.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:42:35.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:35.103 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 1 -- 172.21.15.57:0/2192467053 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1de004f730 con 0x7f1dd8029eb0 2024-03-29T10:42:35.103 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:35.103 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:35.103 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 10 monclient: dump: 2024-03-29T10:42:35.103 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:35.103 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:35.103 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:35.103 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:35.103 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:35.103 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:35.103 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:35.103 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:35.103 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:35.104 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:35.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1de66dc640 5 monclient: authenticate success, global_id 4326 2024-03-29T10:42:35.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1de66dc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:35.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 1 -- 172.21.15.57:0/2192467053 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1de0075f10 con 0x7f1dd8029eb0 2024-03-29T10:42:35.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:35.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 1 -- 172.21.15.57:0/2192467053 <== mon.0 v1:172.21.15.57:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1de00758b0 con 0x7f1dd8029eb0 2024-03-29T10:42:35.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:35.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:35.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.118+0000 7f1dd7fff640 10 monclient: dump: 2024-03-29T10:42:35.105 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:35.105 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:35.105 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:35.105 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:35.105 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:35.105 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:35.105 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:35.105 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:35.105 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:35.105 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:35.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.119+0000 7f1de66dc640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:35.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.119+0000 7f1de66dc640 4 set_mon_vals no callback set 2024-03-29T10:42:35.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.120+0000 7f1de66dc640 10 monclient: shutdown 2024-03-29T10:42:35.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.120+0000 7f1de66dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:35.106 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.120+0000 7f1de66dc640 1 -- 172.21.15.57:0/2192467053 >> v1:172.21.15.57:6789/0 conn(0x7f1dd8029eb0 legacy=0x7f1dd80cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:35.106 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.124+0000 7f1de66dc640 1 -- 172.21.15.57:0/2192467053 shutdown_connections 2024-03-29T10:42:35.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.124+0000 7f1de66dc640 1 -- 172.21.15.57:0/2192467053 >> 172.21.15.57:0/2192467053 conn(0x7f1dd80058f0 msgr2=0x7f1dd8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:35.113 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.127+0000 7f1de66dc640 1 -- 172.21.15.57:0/2192467053 shutdown_connections 2024-03-29T10:42:35.113 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.128+0000 7f1de66dc640 1 -- 172.21.15.57:0/2192467053 wait complete. 2024-03-29T10:42:35.113 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.129+0000 7f1de66dc640 10 monclient: build_initial_monmap 2024-03-29T10:42:35.113 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.129+0000 7f1de66dc640 10 monclient: monmap: 2024-03-29T10:42:35.113 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:42:35.113 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:35.113 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:42:35.114 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:42:35.114 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:35.114 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:35.114 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:42:35.114 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:42:35.114 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:42:35.114 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:35.114 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.129+0000 7f1de66dc640 1 librados: starting msgr at 2024-03-29T10:42:35.114 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.129+0000 7f1de66dc640 1 librados: starting objecter 2024-03-29T10:42:35.114 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.129+0000 7f1de66dc640 1 Processor -- start 2024-03-29T10:42:35.114 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.129+0000 7f1de66dc640 1 -- start start 2024-03-29T10:42:35.114 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.129+0000 7f1de66dc640 1 librados: setting wanted keys 2024-03-29T10:42:35.114 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.129+0000 7f1de66dc640 1 librados: calling monclient init 2024-03-29T10:42:35.114 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.129+0000 7f1de66dc640 10 monclient: init 2024-03-29T10:42:35.115 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.130+0000 7f1de66dc640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:35.115 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.130+0000 7f1de66dc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:42:35.115 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.130+0000 7f1de66dc640 10 monclient(hunting): picked mon.noname-b con 0x7f1dd8029eb0 addr v1:172.21.15.179:6789/0 2024-03-29T10:42:35.115 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.130+0000 7f1de66dc640 10 monclient(hunting): picked mon.noname-c con 0x7f1dd802b0a0 addr v1:172.21.15.57:6790/0 2024-03-29T10:42:35.115 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.130+0000 7f1de66dc640 10 monclient(hunting): picked mon.noname-a con 0x7f1dd80c9570 addr v1:172.21.15.57:6789/0 2024-03-29T10:42:35.115 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.130+0000 7f1de66dc640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dd802e4e0 con 0x7f1dd80c9570 2024-03-29T10:42:35.115 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.130+0000 7f1de66dc640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dd802d8d0 con 0x7f1dd8029eb0 2024-03-29T10:42:35.115 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.130+0000 7f1de66dc640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dd802e750 con 0x7f1dd802b0a0 2024-03-29T10:42:35.115 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.130+0000 7f1de66dc640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:35.115 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.130+0000 7f1de66dc640 10 monclient(hunting): authenticate will time out at 1107.306649s 2024-03-29T10:42:35.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.134+0000 7f1de4ed9640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f1dd802b0a0 0x7f1dd8182f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:33383/0 (socket says 172.21.15.57:33383) 2024-03-29T10:42:35.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.134+0000 7f1de4ed9640 1 -- 172.21.15.57:0/3687753475 learned_addr learned my addr 172.21.15.57:0/3687753475 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:42:35.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.135+0000 7f1dd5ffb640 10 client.?.objecter ms_handle_connect 0x7f1dd802b0a0 2024-03-29T10:42:35.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.135+0000 7f1dd5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:35.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.138+0000 7f1dd5ffb640 1 -- 172.21.15.57:0/3687753475 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3729055633 0 0) 0x7f1dd802e750 con 0x7f1dd802b0a0 2024-03-29T10:42:35.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.138+0000 7f1dd5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:42:35.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.138+0000 7f1dd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:35.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.138+0000 7f1dd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:35.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.138+0000 7f1dd5ffb640 10 monclient(hunting): my global_id is 4346 2024-03-29T10:42:35.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.138+0000 7f1dd5ffb640 1 -- 172.21.15.57:0/3687753475 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dc4004eb0 con 0x7f1dd802b0a0 2024-03-29T10:42:35.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.138+0000 7f1dd5ffb640 1 -- 172.21.15.57:0/3687753475 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1077649132 0 0) 0x7f1dc4004eb0 con 0x7f1dd802b0a0 2024-03-29T10:42:35.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.138+0000 7f1dd5ffb640 1 -- 172.21.15.57:0/3687753475 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1dd802e750 con 0x7f1dd802b0a0 2024-03-29T10:42:35.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.138+0000 7f1dd5ffb640 1 -- 172.21.15.57:0/3687753475 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1dd0001a30 con 0x7f1dd802b0a0 2024-03-29T10:42:35.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.138+0000 7f1dd5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:35.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.138+0000 7f1dd5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:35.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.138+0000 7f1dd5ffb640 10 monclient(hunting): dump: 2024-03-29T10:42:35.122 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:35.123 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:35.123 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:35.123 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:35.123 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:35.123 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:35.123 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:35.123 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:35.123 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:35.123 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:35.123 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.138+0000 7f1dd5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:35.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.139+0000 7f1dd5ffb640 1 -- 172.21.15.57:0/3687753475 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2509665877 0 0) 0x7f1dd802e750 con 0x7f1dd802b0a0 2024-03-29T10:42:35.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.139+0000 7f1dd5ffb640 1 -- 172.21.15.57:0/3687753475 >> v1:172.21.15.179:6789/0 conn(0x7f1dd8029eb0 legacy=0x7f1dd8032240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:35.126 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.139+0000 7f1dd5ffb640 1 -- 172.21.15.57:0/3687753475 >> v1:172.21.15.57:6789/0 conn(0x7f1dd80c9570 legacy=0x7f1dd81836a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:35.126 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.139+0000 7f1dd5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:42:35.126 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.139+0000 7f1dd5ffb640 1 monclient: found mon.c 2024-03-29T10:42:35.126 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.139+0000 7f1dd5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:35.126 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.139+0000 7f1dd5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:42:35.126 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.139+0000 7f1dd5ffb640 1 -- 172.21.15.57:0/3687753475 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd802ea30 con 0x7f1dd802b0a0 2024-03-29T10:42:35.126 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.139+0000 7f1dd5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:42:35.126 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.139+0000 7f1dd5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:42:35.126 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.139+0000 7f1dd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:35.126 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.139+0000 7f1de66dc640 5 monclient: authenticate success, global_id 4346 2024-03-29T10:42:35.126 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.139+0000 7f1de66dc640 10 monclient: _renew_subs 2024-03-29T10:42:35.126 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.139+0000 7f1de66dc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:42:35.126 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.139+0000 7f1de66dc640 1 -- 172.21.15.57:0/3687753475 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dd802cc70 con 0x7f1dd802b0a0 2024-03-29T10:42:35.126 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.139+0000 7f1de66dc640 10 client.4346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:35.127 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.139+0000 7f1de66dc640 10 monclient: _renew_subs 2024-03-29T10:42:35.127 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.139+0000 7f1de66dc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:42:35.127 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.139+0000 7f1de66dc640 1 -- 172.21.15.57:0/3687753475 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dd802d170 con 0x7f1dd802b0a0 2024-03-29T10:42:35.127 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.139+0000 7f1de66dc640 1 librados: init done 2024-03-29T10:42:35.127 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.141+0000 7f1dd5ffb640 1 -- 172.21.15.57:0/3687753475 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1dd00038c0 con 0x7f1dd802b0a0 2024-03-29T10:42:35.127 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.141+0000 7f1dd5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:35.127 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.141+0000 7f1dd77fe640 4 set_mon_vals no callback set 2024-03-29T10:42:35.127 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.142+0000 7f1dcb7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:35.127 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.142+0000 7f1dd5ffb640 1 -- 172.21.15.57:0/3687753475 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1dd0003c50 con 0x7f1dd802b0a0 2024-03-29T10:42:35.127 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.142+0000 7f1dd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:35.127 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.142+0000 7f1dd5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:35.127 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.142+0000 7f1dd5ffb640 10 monclient: dump: 2024-03-29T10:42:35.127 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:35.127 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:35.127 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:35.128 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:35.128 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:35.128 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:35.128 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:35.128 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:35.128 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:35.128 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:35.128 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.142+0000 7f1dcb7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:35.128 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.142+0000 7f1dcb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:42:35.128 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.142+0000 7f1dcb7fe640 1 -- 172.21.15.57:0/3687753475 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dc0005080 con 0x7f1dd802b0a0 2024-03-29T10:42:35.132 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.143+0000 7f1dd5ffb640 1 -- 172.21.15.57:0/3687753475 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f1dd0003e70 con 0x7f1dd802b0a0 2024-03-29T10:42:35.132 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.144+0000 7f1dd5ffb640 1 -- 172.21.15.57:0/3687753475 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f1dd0084c50 con 0x7f1dd802b0a0 2024-03-29T10:42:35.132 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.144+0000 7f1dd5ffb640 10 client.4346.objecter ms_dispatch 0x7f1de0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:35.132 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.144+0000 7f1dd5ffb640 3 client.4346.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:35.132 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.144+0000 7f1de56da640 10 monclient: get_auth_request con 0x7f1dc406dc30 auth_method 0 2024-03-29T10:42:35.132 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.144+0000 7f1dd5ffb640 3 client.4346.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:35.133 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.144+0000 7f1dd5ffb640 20 client.4346.objecter dump_active .. 0 homeless 2024-03-29T10:42:35.133 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.144+0000 7f1dd5ffb640 10 client.4346.objecter ms_handle_connect 0x7f1dc406dc30 2024-03-29T10:42:35.135 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.152+0000 7f1dd5ffb640 1 -- 172.21.15.57:0/3687753475 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1dd0050530 con 0x7f1dd802b0a0 2024-03-29T10:42:35.135 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.152+0000 7f1dd5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:35.135 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.152+0000 7f1dd5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:35.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.490+0000 7f1dcb7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:35.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.490+0000 7f1dcb7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:35.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.490+0000 7f1dcb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:42:35.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.490+0000 7f1dcb7fe640 1 -- 172.21.15.57:0/3687753475 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1dc0005330 con 0x7f1dd802b0a0 2024-03-29T10:42:35.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.491+0000 7f1dd5ffb640 1 -- 172.21.15.57:0/3687753475 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f1dd0052460 con 0x7f1dd802b0a0 2024-03-29T10:42:35.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.491+0000 7f1dd5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:35.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.491+0000 7f1dd5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:35.474 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:42:35.475 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:35.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.494+0000 7f1dcb7fe640 10 librados: watch_flush enter 2024-03-29T10:42:35.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.494+0000 7f1dcb7fe640 10 librados: watch_flush exit 2024-03-29T10:42:35.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.494+0000 7f1dcb7fe640 20 client.4346.objecter shutdown clearing up homeless session... 2024-03-29T10:42:35.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.494+0000 7f1dcb7fe640 10 client.4346.objecter successfully canceled tick 2024-03-29T10:42:35.482 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.494+0000 7f1dcb7fe640 1 -- 172.21.15.57:0/3687753475 >> v1:172.21.15.179:6816/2698295936 conn(0x7f1dc406dc30 legacy=0x7f1dc40700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:35.482 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.494+0000 7f1dcb7fe640 10 monclient: shutdown 2024-03-29T10:42:35.482 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.494+0000 7f1dcb7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:35.482 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.494+0000 7f1dcb7fe640 1 -- 172.21.15.57:0/3687753475 >> v1:172.21.15.57:6790/0 conn(0x7f1dd802b0a0 legacy=0x7f1dd8182f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:35.482 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.496+0000 7f1dcb7fe640 1 -- 172.21.15.57:0/3687753475 shutdown_connections 2024-03-29T10:42:35.482 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.496+0000 7f1dcb7fe640 1 -- 172.21.15.57:0/3687753475 >> 172.21.15.57:0/3687753475 conn(0x7f1dd80058f0 msgr2=0x7f1dd80384d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:35.482 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.497+0000 7f1dcb7fe640 1 -- 172.21.15.57:0/3687753475 shutdown_connections 2024-03-29T10:42:35.482 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.497+0000 7f1dcb7fe640 1 -- 172.21.15.57:0/3687753475 wait complete. 2024-03-29T10:42:35.482 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:35.497+0000 7f1dcb7fe640 1 librados: shutdown 2024-03-29T10:42:35.498 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:42.305 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:42.462 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.479+0000 7f961ab82640 10 monclient: get_monmap_and_config 2024-03-29T10:42:42.462 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.479+0000 7f961ab82640 10 monclient: build_initial_monmap 2024-03-29T10:42:42.462 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.479+0000 7f961ab82640 10 monclient: monmap: 2024-03-29T10:42:42.462 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:42:42.463 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:42.463 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:42:42.479985+0000 2024-03-29T10:42:42.463 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:42:42.479985+0000 2024-03-29T10:42:42.463 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:42.463 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:42.463 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:42:42.463 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:42:42.463 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:42:42.463 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:42.464 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.481+0000 7f961ab82640 1 Processor -- start 2024-03-29T10:42:42.464 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.481+0000 7f961ab82640 1 -- start start 2024-03-29T10:42:42.464 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.481+0000 7f961ab82640 10 monclient: init 2024-03-29T10:42:42.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.482+0000 7f961ab82640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:42.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.482+0000 7f961ab82640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:42:42.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.482+0000 7f961ab82640 10 monclient(hunting): picked mon.noname-c con 0x7f96141457f0 addr v1:172.21.15.179:6789/0 2024-03-29T10:42:42.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.482+0000 7f961ab82640 10 monclient(hunting): picked mon.noname-b con 0x7f961413f650 addr v1:172.21.15.57:6790/0 2024-03-29T10:42:42.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.482+0000 7f961ab82640 10 monclient(hunting): picked mon.noname-a con 0x7f96140a9d70 addr v1:172.21.15.57:6789/0 2024-03-29T10:42:42.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.482+0000 7f961ab82640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96140aaef0 con 0x7f96140a9d70 2024-03-29T10:42:42.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.482+0000 7f961ab82640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9614145bd0 con 0x7f96141457f0 2024-03-29T10:42:42.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.482+0000 7f961ab82640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9614145eb0 con 0x7f961413f650 2024-03-29T10:42:42.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.482+0000 7f961ab82640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:42.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.482+0000 7f961ab82640 10 monclient(hunting): authenticate will time out at 1114.658375s 2024-03-29T10:42:42.470 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.487+0000 7f961911f640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f96140a9d70 0x7f961414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:42417/0 (socket says 172.21.15.57:42417) 2024-03-29T10:42:42.470 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.487+0000 7f961911f640 1 -- 172.21.15.57:0/2119019056 learned_addr learned my addr 172.21.15.57:0/2119019056 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:42:42.471 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.488+0000 7f96137fe640 1 -- 172.21.15.57:0/2119019056 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169207921 0 0) 0x7f96140aaef0 con 0x7f96140a9d70 2024-03-29T10:42:42.472 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.488+0000 7f96137fe640 10 monclient(hunting): _negotiate 2024-03-29T10:42:42.472 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.488+0000 7f96137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:42.472 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.488+0000 7f96137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:42.472 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.488+0000 7f96137fe640 10 monclient(hunting): my global_id is 4329 2024-03-29T10:42:42.472 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.488+0000 7f96137fe640 1 -- 172.21.15.57:0/2119019056 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95fc003520 con 0x7f96140a9d70 2024-03-29T10:42:42.472 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.488+0000 7f96137fe640 1 -- 172.21.15.57:0/2119019056 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 106225400 0 0) 0x7f95fc003520 con 0x7f96140a9d70 2024-03-29T10:42:42.472 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.488+0000 7f96137fe640 1 -- 172.21.15.57:0/2119019056 >> v1:172.21.15.57:6790/0 conn(0x7f961413f650 legacy=0x7f96140b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:42:42.472 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.488+0000 7f96137fe640 1 -- 172.21.15.57:0/2119019056 >> v1:172.21.15.179:6789/0 conn(0x7f96141457f0 legacy=0x7f961413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:42:42.472 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.488+0000 7f96137fe640 10 monclient: _finish_hunting 0 2024-03-29T10:42:42.472 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.488+0000 7f96137fe640 1 monclient: found mon.noname-a 2024-03-29T10:42:42.472 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.488+0000 7f96137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:42.472 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.488+0000 7f96137fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:42:42.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.488+0000 7f96137fe640 1 -- 172.21.15.57:0/2119019056 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9614146340 con 0x7f96140a9d70 2024-03-29T10:42:42.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.488+0000 7f96137fe640 10 monclient: _finish_auth 0 2024-03-29T10:42:42.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.488+0000 7f96137fe640 10 monclient: _check_auth_tickets 2024-03-29T10:42:42.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.488+0000 7f96137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:42.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.488+0000 7f96137fe640 1 -- 172.21.15.57:0/2119019056 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f960c0019f0 con 0x7f96140a9d70 2024-03-29T10:42:42.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.488+0000 7f96137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:42.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.488+0000 7f96137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:42.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.488+0000 7f96137fe640 10 monclient: dump: 2024-03-29T10:42:42.473 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:42.473 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:42.473 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:42.473 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:42.474 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:42.474 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:42.474 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:42.474 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:42.474 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:42.474 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:42.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.489+0000 7f96137fe640 1 -- 172.21.15.57:0/2119019056 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f960c001c50 con 0x7f96140a9d70 2024-03-29T10:42:42.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.489+0000 7f96137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:42.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.489+0000 7f96137fe640 1 -- 172.21.15.57:0/2119019056 <== mon.0 v1:172.21.15.57:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f960c0019f0 con 0x7f96140a9d70 2024-03-29T10:42:42.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.489+0000 7f96137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:42.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.489+0000 7f96137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:42.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.489+0000 7f96137fe640 10 monclient: dump: 2024-03-29T10:42:42.475 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:42.475 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:42.475 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:42.475 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:42.475 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:42.475 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:42.475 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:42.475 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:42.475 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:42.475 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:42.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.489+0000 7f961ab82640 5 monclient: authenticate success, global_id 4329 2024-03-29T10:42:42.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.489+0000 7f961ab82640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:42.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.489+0000 7f961ab82640 4 set_mon_vals no callback set 2024-03-29T10:42:42.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.490+0000 7f961ab82640 10 monclient: shutdown 2024-03-29T10:42:42.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.490+0000 7f961ab82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:42.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.490+0000 7f961ab82640 1 -- 172.21.15.57:0/2119019056 >> v1:172.21.15.57:6789/0 conn(0x7f96140a9d70 legacy=0x7f961414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:42.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.492+0000 7f961ab82640 1 -- 172.21.15.57:0/2119019056 shutdown_connections 2024-03-29T10:42:42.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.492+0000 7f961ab82640 1 -- 172.21.15.57:0/2119019056 >> 172.21.15.57:0/2119019056 conn(0x7f9614066a00 msgr2=0x7f9614066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:42.477 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.493+0000 7f961ab82640 1 -- 172.21.15.57:0/2119019056 shutdown_connections 2024-03-29T10:42:42.477 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.494+0000 7f961ab82640 1 -- 172.21.15.57:0/2119019056 wait complete. 2024-03-29T10:42:42.478 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.495+0000 7f961ab82640 10 monclient: build_initial_monmap 2024-03-29T10:42:42.478 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.495+0000 7f961ab82640 10 monclient: monmap: 2024-03-29T10:42:42.478 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:42:42.478 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:42.478 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:42:42.479 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:42:42.479 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:42.479 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:42.479 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:42:42.479 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:42:42.479 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:42:42.479 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:42.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.496+0000 7f961ab82640 1 librados: starting msgr at 2024-03-29T10:42:42.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.496+0000 7f961ab82640 1 librados: starting objecter 2024-03-29T10:42:42.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.496+0000 7f961ab82640 1 Processor -- start 2024-03-29T10:42:42.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.496+0000 7f961ab82640 1 -- start start 2024-03-29T10:42:42.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.496+0000 7f961ab82640 1 librados: setting wanted keys 2024-03-29T10:42:42.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.496+0000 7f961ab82640 1 librados: calling monclient init 2024-03-29T10:42:42.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.496+0000 7f961ab82640 10 monclient: init 2024-03-29T10:42:42.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.497+0000 7f961ab82640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:42.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.497+0000 7f961ab82640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:42:42.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.497+0000 7f961ab82640 10 monclient(hunting): picked mon.noname-b con 0x7f96140a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T10:42:42.482 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.497+0000 7f961ab82640 10 monclient(hunting): picked mon.noname-c con 0x7f961413f650 addr v1:172.21.15.57:6790/0 2024-03-29T10:42:42.482 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.497+0000 7f961ab82640 10 monclient(hunting): picked mon.noname-a con 0x7f96141457f0 addr v1:172.21.15.57:6789/0 2024-03-29T10:42:42.482 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.497+0000 7f961ab82640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96141418c0 con 0x7f96141457f0 2024-03-29T10:42:42.482 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.497+0000 7f961ab82640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f961413bd50 con 0x7f96140a9d70 2024-03-29T10:42:42.482 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.497+0000 7f961ab82640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f961413bee0 con 0x7f961413f650 2024-03-29T10:42:42.482 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.497+0000 7f961ab82640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:42.482 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.497+0000 7f961ab82640 10 monclient(hunting): authenticate will time out at 1114.673806s 2024-03-29T10:42:42.485 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.501+0000 7f961891e640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f96140a9d70 0x7f9614140180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:55895/0 (socket says 172.21.15.57:55895) 2024-03-29T10:42:42.485 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.501+0000 7f961891e640 1 -- 172.21.15.57:0/2747660646 learned_addr learned my addr 172.21.15.57:0/2747660646 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:42:42.487 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.504+0000 7f96117fa640 10 client.?.objecter ms_handle_connect 0x7f96140a9d70 2024-03-29T10:42:42.491 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.504+0000 7f96117fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:42.491 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.504+0000 7f96117fa640 10 client.?.objecter ms_handle_connect 0x7f961413f650 2024-03-29T10:42:42.491 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.504+0000 7f96117fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:42.491 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.504+0000 7f96117fa640 1 -- 172.21.15.57:0/2747660646 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658697428 0 0) 0x7f961413bd50 con 0x7f96140a9d70 2024-03-29T10:42:42.491 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.504+0000 7f96117fa640 10 monclient(hunting): _negotiate 2024-03-29T10:42:42.491 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.504+0000 7f96117fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:42.492 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.504+0000 7f96117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:42.492 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.504+0000 7f96117fa640 10 monclient(hunting): my global_id is 4342 2024-03-29T10:42:42.492 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.504+0000 7f96117fa640 1 -- 172.21.15.57:0/2747660646 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9600004eb0 con 0x7f96140a9d70 2024-03-29T10:42:42.492 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.505+0000 7f96117fa640 1 -- 172.21.15.57:0/2747660646 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2278783802 0 0) 0x7f9600004eb0 con 0x7f96140a9d70 2024-03-29T10:42:42.492 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.505+0000 7f96117fa640 1 -- 172.21.15.57:0/2747660646 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f961413bd50 con 0x7f96140a9d70 2024-03-29T10:42:42.492 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.505+0000 7f96117fa640 1 -- 172.21.15.57:0/2747660646 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f96040017d0 con 0x7f96140a9d70 2024-03-29T10:42:42.492 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.505+0000 7f96117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:42.492 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.505+0000 7f96117fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:42.492 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.505+0000 7f96117fa640 10 monclient(hunting): dump: 2024-03-29T10:42:42.492 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:42.492 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:42.492 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:42.492 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:42.493 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:42.493 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:42.493 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:42.493 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:42.493 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:42.493 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:42.493 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.505+0000 7f96117fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:42.493 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.505+0000 7f96117fa640 1 -- 172.21.15.57:0/2747660646 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1214904118 0 0) 0x7f961413bee0 con 0x7f961413f650 2024-03-29T10:42:42.493 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.505+0000 7f96117fa640 10 monclient(hunting): _negotiate 2024-03-29T10:42:42.493 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.505+0000 7f96117fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:42.493 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.505+0000 7f96117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:42.493 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.505+0000 7f96117fa640 10 monclient(hunting): my global_id is 4349 2024-03-29T10:42:42.493 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.505+0000 7f96117fa640 1 -- 172.21.15.57:0/2747660646 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9600004eb0 con 0x7f961413f650 2024-03-29T10:42:42.494 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.505+0000 7f96117fa640 1 -- 172.21.15.57:0/2747660646 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1000750072 0 0) 0x7f961413bd50 con 0x7f96140a9d70 2024-03-29T10:42:42.494 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.505+0000 7f96117fa640 1 -- 172.21.15.57:0/2747660646 >> v1:172.21.15.57:6790/0 conn(0x7f961413f650 legacy=0x7f96141dedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:42.494 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.506+0000 7f96117fa640 1 -- 172.21.15.57:0/2747660646 >> v1:172.21.15.57:6789/0 conn(0x7f96141457f0 legacy=0x7f96141e2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:42.494 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.506+0000 7f96117fa640 10 monclient: _finish_hunting 0 2024-03-29T10:42:42.494 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.506+0000 7f96117fa640 1 monclient: found mon.b 2024-03-29T10:42:42.494 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.506+0000 7f96117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:42.494 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.506+0000 7f96117fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:42:42.494 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.506+0000 7f96117fa640 1 -- 172.21.15.57:0/2747660646 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9614135a00 con 0x7f96140a9d70 2024-03-29T10:42:42.494 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.506+0000 7f96117fa640 10 monclient: _finish_auth 0 2024-03-29T10:42:42.494 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.506+0000 7f96117fa640 10 monclient: _check_auth_tickets 2024-03-29T10:42:42.494 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.506+0000 7f96117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:42.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.506+0000 7f961ab82640 5 monclient: authenticate success, global_id 4342 2024-03-29T10:42:42.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.506+0000 7f961ab82640 10 monclient: _renew_subs 2024-03-29T10:42:42.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.506+0000 7f961ab82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:42:42.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.506+0000 7f961ab82640 1 -- 172.21.15.57:0/2747660646 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9614135c70 con 0x7f96140a9d70 2024-03-29T10:42:42.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.506+0000 7f961ab82640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:42.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.506+0000 7f961ab82640 10 monclient: _renew_subs 2024-03-29T10:42:42.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.506+0000 7f961ab82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:42:42.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.506+0000 7f961ab82640 1 -- 172.21.15.57:0/2747660646 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f961413e140 con 0x7f96140a9d70 2024-03-29T10:42:42.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.506+0000 7f961ab82640 1 librados: init done 2024-03-29T10:42:42.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.506+0000 7f96117fa640 1 -- 172.21.15.57:0/2747660646 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9604001d60 con 0x7f96140a9d70 2024-03-29T10:42:42.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.506+0000 7f96117fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:42.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.506+0000 7f96117fa640 1 -- 172.21.15.57:0/2747660646 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f9604004a00 con 0x7f96140a9d70 2024-03-29T10:42:42.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.506+0000 7f96117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:42.496 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.506+0000 7f96117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:42.496 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.506+0000 7f96117fa640 10 monclient: dump: 2024-03-29T10:42:42.496 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:42.496 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:42.496 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:42.496 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:42.496 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:42.496 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:42.496 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:42.496 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:42.496 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:42.496 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:42.498 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.508+0000 7f96127fc640 4 set_mon_vals no callback set 2024-03-29T10:42:42.498 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.510+0000 7f961ab82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:42.498 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.510+0000 7f961ab82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:42.498 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.510+0000 7f961ab82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:42:42.498 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.510+0000 7f961ab82640 1 -- 172.21.15.57:0/2747660646 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95f4005080 con 0x7f96140a9d70 2024-03-29T10:42:42.498 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.511+0000 7f96117fa640 1 -- 172.21.15.57:0/2747660646 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f9604004b60 con 0x7f96140a9d70 2024-03-29T10:42:42.498 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.512+0000 7f96117fa640 1 -- 172.21.15.57:0/2747660646 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f9604084a80 con 0x7f96140a9d70 2024-03-29T10:42:42.498 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.513+0000 7f96117fa640 10 client.4342.objecter ms_dispatch 0x7f9614000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:42.499 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.513+0000 7f96117fa640 3 client.4342.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:42.499 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.513+0000 7f96117fa640 3 client.4342.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:42.499 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.513+0000 7f96117fa640 20 client.4342.objecter dump_active .. 0 homeless 2024-03-29T10:42:42.499 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.514+0000 7f9613fff640 10 monclient: get_auth_request con 0x7f960006dd60 auth_method 0 2024-03-29T10:42:42.499 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.514+0000 7f96117fa640 10 client.4342.objecter ms_handle_connect 0x7f960006dd60 2024-03-29T10:42:42.504 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.520+0000 7f96117fa640 1 -- 172.21.15.57:0/2747660646 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9604050190 con 0x7f96140a9d70 2024-03-29T10:42:42.504 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.520+0000 7f96117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:42.504 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.520+0000 7f96117fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:42.840 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.855+0000 7f961ab82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:42.840 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.855+0000 7f961ab82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:42.840 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.855+0000 7f961ab82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:42:42.840 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.855+0000 7f961ab82640 1 -- 172.21.15.57:0/2747660646 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95f4005330 con 0x7f96140a9d70 2024-03-29T10:42:42.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.859+0000 7f96117fa640 1 -- 172.21.15.57:0/2747660646 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f9604052290 con 0x7f96140a9d70 2024-03-29T10:42:42.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.859+0000 7f96117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:42.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.859+0000 7f96117fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:42.842 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:42:42.842 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:42.846 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.862+0000 7f961ab82640 10 librados: watch_flush enter 2024-03-29T10:42:42.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.863+0000 7f961ab82640 10 librados: watch_flush exit 2024-03-29T10:42:42.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.863+0000 7f961ab82640 20 client.4342.objecter shutdown clearing up homeless session... 2024-03-29T10:42:42.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.863+0000 7f961ab82640 10 client.4342.objecter successfully canceled tick 2024-03-29T10:42:42.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.863+0000 7f961ab82640 1 -- 172.21.15.57:0/2747660646 >> v1:172.21.15.179:6816/2698295936 conn(0x7f960006dd60 legacy=0x7f9600070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:42.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.863+0000 7f961ab82640 10 monclient: shutdown 2024-03-29T10:42:42.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.863+0000 7f961ab82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:42.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.863+0000 7f961ab82640 1 -- 172.21.15.57:0/2747660646 >> v1:172.21.15.179:6789/0 conn(0x7f96140a9d70 legacy=0x7f9614140180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:42.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.863+0000 7f961ab82640 1 -- 172.21.15.57:0/2747660646 shutdown_connections 2024-03-29T10:42:42.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.863+0000 7f961ab82640 1 -- 172.21.15.57:0/2747660646 >> 172.21.15.57:0/2747660646 conn(0x7f9614066a00 msgr2=0x7f96140b60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:42.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.868+0000 7f961ab82640 1 -- 172.21.15.57:0/2747660646 shutdown_connections 2024-03-29T10:42:42.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.869+0000 7f961ab82640 1 -- 172.21.15.57:0/2747660646 wait complete. 2024-03-29T10:42:42.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:42.869+0000 7f961ab82640 1 librados: shutdown 2024-03-29T10:42:42.868 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:49.676 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:49.819 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.836+0000 7f58e8fa8640 10 monclient: get_monmap_and_config 2024-03-29T10:42:49.819 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.836+0000 7f58e8fa8640 10 monclient: build_initial_monmap 2024-03-29T10:42:49.819 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.836+0000 7f58e8fa8640 10 monclient: monmap: 2024-03-29T10:42:49.819 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:42:49.819 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:49.820 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:42:49.837140+0000 2024-03-29T10:42:49.820 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:42:49.837140+0000 2024-03-29T10:42:49.820 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:49.820 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:49.820 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:42:49.820 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:42:49.820 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:42:49.820 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:49.821 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.838+0000 7f58e8fa8640 1 Processor -- start 2024-03-29T10:42:49.821 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.838+0000 7f58e8fa8640 1 -- start start 2024-03-29T10:42:49.822 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.838+0000 7f58e8fa8640 10 monclient: init 2024-03-29T10:42:49.822 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.839+0000 7f58e8fa8640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:49.822 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.839+0000 7f58e8fa8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:42:49.822 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.840+0000 7f58e8fa8640 10 monclient(hunting): picked mon.noname-a con 0x7f58e41457f0 addr v1:172.21.15.57:6789/0 2024-03-29T10:42:49.823 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.840+0000 7f58e8fa8640 10 monclient(hunting): picked mon.noname-b con 0x7f58e413f650 addr v1:172.21.15.57:6790/0 2024-03-29T10:42:49.823 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.840+0000 7f58e8fa8640 10 monclient(hunting): picked mon.noname-c con 0x7f58e40a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T10:42:49.823 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.840+0000 7f58e8fa8640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e40aaef0 con 0x7f58e41457f0 2024-03-29T10:42:49.823 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.840+0000 7f58e8fa8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e4145bd0 con 0x7f58e40a9d70 2024-03-29T10:42:49.823 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.840+0000 7f58e8fa8640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e4145eb0 con 0x7f58e413f650 2024-03-29T10:42:49.823 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.840+0000 7f58e8fa8640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:49.823 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.840+0000 7f58e8fa8640 10 monclient(hunting): authenticate will time out at 1122.016109s 2024-03-29T10:42:49.823 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.840+0000 7f58e1d9b640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f58e413f650 0x7f58e40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:57182/0 (socket says 172.21.15.57:57182) 2024-03-29T10:42:49.824 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.840+0000 7f58e1d9b640 1 -- 172.21.15.57:0/4219728820 learned_addr learned my addr 172.21.15.57:0/4219728820 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:42:49.826 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.843+0000 7f58e159a640 1 -- 172.21.15.57:0/4219728820 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1646932642 0 0) 0x7f58e4145eb0 con 0x7f58e413f650 2024-03-29T10:42:49.826 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.843+0000 7f58e159a640 10 monclient(hunting): _negotiate 2024-03-29T10:42:49.826 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.843+0000 7f58e159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:49.826 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.843+0000 7f58e159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:49.826 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.843+0000 7f58e159a640 10 monclient(hunting): my global_id is 4352 2024-03-29T10:42:49.826 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.843+0000 7f58e159a640 1 -- 172.21.15.57:0/4219728820 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58cc003520 con 0x7f58e413f650 2024-03-29T10:42:49.829 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.843+0000 7f58e159a640 1 -- 172.21.15.57:0/4219728820 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2606363845 0 0) 0x7f58cc003520 con 0x7f58e413f650 2024-03-29T10:42:49.829 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.844+0000 7f58e159a640 1 -- 172.21.15.57:0/4219728820 >> v1:172.21.15.179:6789/0 conn(0x7f58e40a9d70 legacy=0x7f58e414d8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:42:49.829 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.844+0000 7f58e159a640 1 -- 172.21.15.57:0/4219728820 >> v1:172.21.15.57:6789/0 conn(0x7f58e41457f0 legacy=0x7f58e413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:42:49.829 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.844+0000 7f58e159a640 10 monclient: _finish_hunting 0 2024-03-29T10:42:49.829 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.844+0000 7f58e159a640 1 monclient: found mon.noname-b 2024-03-29T10:42:49.829 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.844+0000 7f58e159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:49.829 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.844+0000 7f58e159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:42:49.829 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.844+0000 7f58e159a640 1 -- 172.21.15.57:0/4219728820 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e4146340 con 0x7f58e413f650 2024-03-29T10:42:49.829 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.844+0000 7f58e159a640 10 monclient: _finish_auth 0 2024-03-29T10:42:49.829 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.844+0000 7f58e159a640 10 monclient: _check_auth_tickets 2024-03-29T10:42:49.830 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.844+0000 7f58e159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:49.830 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.844+0000 7f58e8fa8640 5 monclient: authenticate success, global_id 4352 2024-03-29T10:42:49.830 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.844+0000 7f58e8fa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:49.830 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.844+0000 7f58e159a640 1 -- 172.21.15.57:0/4219728820 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f58dc001a10 con 0x7f58e413f650 2024-03-29T10:42:49.830 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.844+0000 7f58e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:49.830 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.844+0000 7f58e159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:49.830 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.844+0000 7f58e159a640 10 monclient: dump: 2024-03-29T10:42:49.830 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:49.830 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:49.830 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:49.830 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:49.830 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:49.830 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:49.830 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:49.831 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:49.831 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:49.831 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:49.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.844+0000 7f58e8fa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:49.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.844+0000 7f58e159a640 1 -- 172.21.15.57:0/4219728820 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58dc001ca0 con 0x7f58e413f650 2024-03-29T10:42:49.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.844+0000 7f58e159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:49.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.844+0000 7f58e8fa8640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:49.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.844+0000 7f58e8fa8640 4 set_mon_vals no callback set 2024-03-29T10:42:49.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.845+0000 7f58e8fa8640 10 monclient: shutdown 2024-03-29T10:42:49.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.845+0000 7f58e8fa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:49.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.845+0000 7f58e8fa8640 1 -- 172.21.15.57:0/4219728820 >> v1:172.21.15.57:6790/0 conn(0x7f58e413f650 legacy=0x7f58e40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:49.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.846+0000 7f58e8fa8640 1 -- 172.21.15.57:0/4219728820 shutdown_connections 2024-03-29T10:42:49.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.846+0000 7f58e8fa8640 1 -- 172.21.15.57:0/4219728820 >> 172.21.15.57:0/4219728820 conn(0x7f58e4066a00 msgr2=0x7f58e4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:49.832 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.847+0000 7f58e8fa8640 1 -- 172.21.15.57:0/4219728820 shutdown_connections 2024-03-29T10:42:49.832 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.847+0000 7f58e8fa8640 1 -- 172.21.15.57:0/4219728820 wait complete. 2024-03-29T10:42:49.832 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.848+0000 7f58e8fa8640 10 monclient: build_initial_monmap 2024-03-29T10:42:49.832 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.848+0000 7f58e8fa8640 10 monclient: monmap: 2024-03-29T10:42:49.832 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:42:49.832 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:49.832 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:42:49.832 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:42:49.833 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:49.833 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:49.833 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:42:49.833 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:42:49.833 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:42:49.833 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:49.833 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.848+0000 7f58e8fa8640 1 librados: starting msgr at 2024-03-29T10:42:49.833 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.848+0000 7f58e8fa8640 1 librados: starting objecter 2024-03-29T10:42:49.833 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.849+0000 7f58e8fa8640 1 Processor -- start 2024-03-29T10:42:49.833 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.849+0000 7f58e8fa8640 1 -- start start 2024-03-29T10:42:49.833 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.849+0000 7f58e8fa8640 1 librados: setting wanted keys 2024-03-29T10:42:49.833 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.849+0000 7f58e8fa8640 1 librados: calling monclient init 2024-03-29T10:42:49.833 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.849+0000 7f58e8fa8640 10 monclient: init 2024-03-29T10:42:49.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.849+0000 7f58e8fa8640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:49.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.850+0000 7f58e8fa8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:42:49.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.850+0000 7f58e8fa8640 10 monclient(hunting): picked mon.noname-b con 0x7f58e40a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T10:42:49.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.850+0000 7f58e8fa8640 10 monclient(hunting): picked mon.noname-c con 0x7f58e413f650 addr v1:172.21.15.57:6790/0 2024-03-29T10:42:49.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.850+0000 7f58e8fa8640 10 monclient(hunting): picked mon.noname-a con 0x7f58e41457f0 addr v1:172.21.15.57:6789/0 2024-03-29T10:42:49.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.850+0000 7f58e8fa8640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e41427c0 con 0x7f58e41457f0 2024-03-29T10:42:49.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.850+0000 7f58e8fa8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e4142430 con 0x7f58e40a9d70 2024-03-29T10:42:49.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.850+0000 7f58e8fa8640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e4142c50 con 0x7f58e413f650 2024-03-29T10:42:49.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.850+0000 7f58e8fa8640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:49.835 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.850+0000 7f58e8fa8640 10 monclient(hunting): authenticate will time out at 1122.026121s 2024-03-29T10:42:49.835 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.852+0000 7f58e1d9b640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f58e413f650 0x7f58e4136f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:34362/0 (socket says 172.21.15.57:34362) 2024-03-29T10:42:49.835 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.852+0000 7f58e1d9b640 1 -- 172.21.15.57:0/2888261152 learned_addr learned my addr 172.21.15.57:0/2888261152 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:42:49.835 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.853+0000 7f58d2ffd640 10 client.?.objecter ms_handle_connect 0x7f58e41457f0 2024-03-29T10:42:49.835 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.853+0000 7f58d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:49.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.853+0000 7f58d2ffd640 1 -- 172.21.15.57:0/2888261152 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199050544 0 0) 0x7f58e41427c0 con 0x7f58e41457f0 2024-03-29T10:42:49.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.853+0000 7f58d2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:49.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.853+0000 7f58d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:49.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.853+0000 7f58d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:49.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.853+0000 7f58d2ffd640 10 monclient(hunting): my global_id is 4332 2024-03-29T10:42:49.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.853+0000 7f58d2ffd640 1 -- 172.21.15.57:0/2888261152 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58c4004eb0 con 0x7f58e41457f0 2024-03-29T10:42:49.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.853+0000 7f58d2ffd640 10 client.?.objecter ms_handle_connect 0x7f58e413f650 2024-03-29T10:42:49.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.853+0000 7f58d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:49.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.853+0000 7f58d2ffd640 1 -- 172.21.15.57:0/2888261152 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2382997926 0 0) 0x7f58c4004eb0 con 0x7f58e41457f0 2024-03-29T10:42:49.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.853+0000 7f58d2ffd640 1 -- 172.21.15.57:0/2888261152 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58e41427c0 con 0x7f58e41457f0 2024-03-29T10:42:49.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.854+0000 7f58d2ffd640 1 -- 172.21.15.57:0/2888261152 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f58d40017f0 con 0x7f58e41457f0 2024-03-29T10:42:49.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.854+0000 7f58d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:49.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.854+0000 7f58d2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:49.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.854+0000 7f58d2ffd640 10 monclient(hunting): dump: 2024-03-29T10:42:49.839 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:49.839 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:49.839 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:49.839 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:49.839 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:49.839 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:49.839 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:49.839 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:49.839 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:49.839 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:49.839 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.854+0000 7f58d2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:49.839 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.854+0000 7f58d2ffd640 1 -- 172.21.15.57:0/2888261152 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4232200490 0 0) 0x7f58e4142c50 con 0x7f58e413f650 2024-03-29T10:42:49.839 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.854+0000 7f58d2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:49.839 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.854+0000 7f58d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:49.839 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.854+0000 7f58d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:49.839 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.854+0000 7f58d2ffd640 10 monclient(hunting): my global_id is 4355 2024-03-29T10:42:49.840 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.854+0000 7f58d2ffd640 1 -- 172.21.15.57:0/2888261152 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58c4004eb0 con 0x7f58e413f650 2024-03-29T10:42:49.840 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.854+0000 7f58d2ffd640 1 -- 172.21.15.57:0/2888261152 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4093872913 0 0) 0x7f58e41427c0 con 0x7f58e41457f0 2024-03-29T10:42:49.840 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.854+0000 7f58d2ffd640 1 -- 172.21.15.57:0/2888261152 >> v1:172.21.15.57:6790/0 conn(0x7f58e413f650 legacy=0x7f58e4136f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:49.840 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.854+0000 7f58d2ffd640 1 -- 172.21.15.57:0/2888261152 >> v1:172.21.15.179:6789/0 conn(0x7f58e40a9d70 legacy=0x7f58e4141d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:49.841 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.858+0000 7f58d2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:49.841 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.858+0000 7f58d2ffd640 1 monclient: found mon.a 2024-03-29T10:42:49.841 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.858+0000 7f58d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:49.841 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.858+0000 7f58d2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:42:49.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.858+0000 7f58d2ffd640 1 -- 172.21.15.57:0/2888261152 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e4142de0 con 0x7f58e41457f0 2024-03-29T10:42:49.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.858+0000 7f58d2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:42:49.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.858+0000 7f58d2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:49.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.858+0000 7f58d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:49.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.858+0000 7f58d2ffd640 1 -- 172.21.15.57:0/2888261152 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58d4001950 con 0x7f58e41457f0 2024-03-29T10:42:49.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.858+0000 7f58d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:49.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.858+0000 7f58d3fff640 4 set_mon_vals no callback set 2024-03-29T10:42:49.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.859+0000 7f58d2ffd640 1 -- 172.21.15.57:0/2888261152 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f58d4003830 con 0x7f58e41457f0 2024-03-29T10:42:49.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.860+0000 7f58e8fa8640 5 monclient: authenticate success, global_id 4332 2024-03-29T10:42:49.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.860+0000 7f58e8fa8640 10 monclient: _renew_subs 2024-03-29T10:42:49.848 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.860+0000 7f58e8fa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:42:49.848 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.860+0000 7f58e8fa8640 1 -- 172.21.15.57:0/2888261152 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e41445c0 con 0x7f58e41457f0 2024-03-29T10:42:49.848 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.860+0000 7f58e8fa8640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:49.848 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.860+0000 7f58e8fa8640 10 monclient: _renew_subs 2024-03-29T10:42:49.848 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.860+0000 7f58e8fa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:42:49.848 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.860+0000 7f58e8fa8640 1 -- 172.21.15.57:0/2888261152 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e4137690 con 0x7f58e41457f0 2024-03-29T10:42:49.848 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.860+0000 7f58e8fa8640 1 librados: init done 2024-03-29T10:42:49.848 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.860+0000 7f58d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:49.848 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.860+0000 7f58d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:49.848 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.860+0000 7f58d2ffd640 10 monclient: dump: 2024-03-29T10:42:49.848 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:49.848 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:49.848 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:49.848 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:49.848 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:49.849 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:49.849 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:49.849 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:49.849 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:49.849 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:49.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.861+0000 7f58e8fa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:49.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.861+0000 7f58e8fa8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:49.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.861+0000 7f58e8fa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:42:49.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.861+0000 7f58e8fa8640 1 -- 172.21.15.57:0/2888261152 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e4075f10 con 0x7f58e41457f0 2024-03-29T10:42:49.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.863+0000 7f58d2ffd640 1 -- 172.21.15.57:0/2888261152 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f58d4019800 con 0x7f58e41457f0 2024-03-29T10:42:49.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.863+0000 7f58e259c640 10 monclient: get_auth_request con 0x7f58c4076570 auth_method 0 2024-03-29T10:42:49.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.863+0000 7f58d2ffd640 1 -- 172.21.15.57:0/2888261152 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f58d40837a0 con 0x7f58e41457f0 2024-03-29T10:42:49.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.863+0000 7f58d2ffd640 10 client.4332.objecter ms_dispatch 0x7f58e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:49.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.863+0000 7f58d2ffd640 3 client.4332.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:49.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.863+0000 7f58d2ffd640 3 client.4332.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:49.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.863+0000 7f58d2ffd640 20 client.4332.objecter dump_active .. 0 homeless 2024-03-29T10:42:49.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.864+0000 7f58d2ffd640 10 client.4332.objecter ms_handle_connect 0x7f58c4076570 2024-03-29T10:42:49.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.871+0000 7f58d2ffd640 1 -- 172.21.15.57:0/2888261152 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f58d404f080 con 0x7f58e41457f0 2024-03-29T10:42:49.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.871+0000 7f58d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:49.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:49.871+0000 7f58d2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:50.203 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:50.220+0000 7f58e8fa8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:50.203 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:50.220+0000 7f58e8fa8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:50.203 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:50.220+0000 7f58e8fa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:42:50.203 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:50.220+0000 7f58e8fa8640 1 -- 172.21.15.57:0/2888261152 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58e40b3680 con 0x7f58e41457f0 2024-03-29T10:42:50.204 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:50.221+0000 7f58d2ffd640 1 -- 172.21.15.57:0/2888261152 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f58d4050fb0 con 0x7f58e41457f0 2024-03-29T10:42:50.204 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:50.221+0000 7f58d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:50.204 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:50.221+0000 7f58d2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:50.208 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:42:50.208 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:50.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:50.225+0000 7f58e8fa8640 10 librados: watch_flush enter 2024-03-29T10:42:50.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:50.225+0000 7f58e8fa8640 10 librados: watch_flush exit 2024-03-29T10:42:50.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:50.225+0000 7f58e8fa8640 20 client.4332.objecter shutdown clearing up homeless session... 2024-03-29T10:42:50.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:50.225+0000 7f58e8fa8640 10 client.4332.objecter successfully canceled tick 2024-03-29T10:42:50.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:50.225+0000 7f58e8fa8640 1 -- 172.21.15.57:0/2888261152 >> v1:172.21.15.179:6816/2698295936 conn(0x7f58c4076570 legacy=0x7f58c4078a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:50.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:50.225+0000 7f58e8fa8640 10 monclient: shutdown 2024-03-29T10:42:50.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:50.225+0000 7f58e8fa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:50.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:50.225+0000 7f58e8fa8640 1 -- 172.21.15.57:0/2888261152 >> v1:172.21.15.57:6789/0 conn(0x7f58e41457f0 legacy=0x7f58e41d6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:50.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:50.226+0000 7f58e8fa8640 1 -- 172.21.15.57:0/2888261152 shutdown_connections 2024-03-29T10:42:50.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:50.226+0000 7f58e8fa8640 1 -- 172.21.15.57:0/2888261152 >> 172.21.15.57:0/2888261152 conn(0x7f58e4066a00 msgr2=0x7f58e40b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:50.210 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:50.226+0000 7f58e8fa8640 1 -- 172.21.15.57:0/2888261152 shutdown_connections 2024-03-29T10:42:50.210 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:50.226+0000 7f58e8fa8640 1 -- 172.21.15.57:0/2888261152 wait complete. 2024-03-29T10:42:50.210 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:50.226+0000 7f58e8fa8640 1 librados: shutdown 2024-03-29T10:42:50.228 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:57.035 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:57.203 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.220+0000 7f1d6b69a640 10 monclient: get_monmap_and_config 2024-03-29T10:42:57.203 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.220+0000 7f1d6b69a640 10 monclient: build_initial_monmap 2024-03-29T10:42:57.204 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.220+0000 7f1d6b69a640 10 monclient: monmap: 2024-03-29T10:42:57.204 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:42:57.204 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:57.204 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:42:57.221279+0000 2024-03-29T10:42:57.204 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:42:57.221279+0000 2024-03-29T10:42:57.204 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:57.204 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:57.204 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:42:57.204 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:42:57.204 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:42:57.204 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:57.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.221+0000 7f1d6b69a640 1 Processor -- start 2024-03-29T10:42:57.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.222+0000 7f1d6b69a640 1 -- start start 2024-03-29T10:42:57.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.222+0000 7f1d6b69a640 10 monclient: init 2024-03-29T10:42:57.206 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.223+0000 7f1d6b69a640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:57.206 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.223+0000 7f1d6b69a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:42:57.206 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.223+0000 7f1d6b69a640 10 monclient(hunting): picked mon.noname-b con 0x7f1d641457f0 addr v1:172.21.15.57:6790/0 2024-03-29T10:42:57.207 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.223+0000 7f1d6b69a640 10 monclient(hunting): picked mon.noname-c con 0x7f1d6413f650 addr v1:172.21.15.179:6789/0 2024-03-29T10:42:57.207 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.223+0000 7f1d6b69a640 10 monclient(hunting): picked mon.noname-a con 0x7f1d640a9d70 addr v1:172.21.15.57:6789/0 2024-03-29T10:42:57.207 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.223+0000 7f1d6b69a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d640aaef0 con 0x7f1d640a9d70 2024-03-29T10:42:57.207 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.223+0000 7f1d6b69a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d64145bd0 con 0x7f1d6413f650 2024-03-29T10:42:57.207 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.223+0000 7f1d6b69a640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d64145eb0 con 0x7f1d641457f0 2024-03-29T10:42:57.207 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.223+0000 7f1d6b69a640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:57.207 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.223+0000 7f1d6b69a640 10 monclient(hunting): authenticate will time out at 1129.399246s 2024-03-29T10:42:57.207 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.223+0000 7f1d69c37640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1d640a9d70 0x7f1d6414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:50392/0 (socket says 172.21.15.57:50392) 2024-03-29T10:42:57.207 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.223+0000 7f1d69c37640 1 -- 172.21.15.57:0/1977639781 learned_addr learned my addr 172.21.15.57:0/1977639781 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:42:57.208 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.224+0000 7f1d5bfff640 1 -- 172.21.15.57:0/1977639781 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300482788 0 0) 0x7f1d64145eb0 con 0x7f1d641457f0 2024-03-29T10:42:57.208 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.224+0000 7f1d5bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:57.208 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.224+0000 7f1d5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:57.208 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.224+0000 7f1d5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:57.208 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.224+0000 7f1d5bfff640 10 monclient(hunting): my global_id is 4358 2024-03-29T10:42:57.208 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.224+0000 7f1d5bfff640 1 -- 172.21.15.57:0/1977639781 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d50003520 con 0x7f1d641457f0 2024-03-29T10:42:57.208 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.224+0000 7f1d5bfff640 1 -- 172.21.15.57:0/1977639781 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2962520048 0 0) 0x7f1d64145bd0 con 0x7f1d6413f650 2024-03-29T10:42:57.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.224+0000 7f1d5bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:57.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.224+0000 7f1d5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:57.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.224+0000 7f1d5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:57.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.224+0000 7f1d5bfff640 10 monclient(hunting): my global_id is 4345 2024-03-29T10:42:57.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.224+0000 7f1d5bfff640 1 -- 172.21.15.57:0/1977639781 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d64145eb0 con 0x7f1d6413f650 2024-03-29T10:42:57.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.225+0000 7f1d5bfff640 1 -- 172.21.15.57:0/1977639781 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 665772205 0 0) 0x7f1d50003520 con 0x7f1d641457f0 2024-03-29T10:42:57.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.225+0000 7f1d5bfff640 1 -- 172.21.15.57:0/1977639781 >> v1:172.21.15.179:6789/0 conn(0x7f1d6413f650 legacy=0x7f1d640b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:57.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.225+0000 7f1d5bfff640 1 -- 172.21.15.57:0/1977639781 >> v1:172.21.15.57:6789/0 conn(0x7f1d640a9d70 legacy=0x7f1d6414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:57.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.225+0000 7f1d5bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:42:57.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.225+0000 7f1d5bfff640 1 monclient: found mon.noname-b 2024-03-29T10:42:57.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.225+0000 7f1d5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:57.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.225+0000 7f1d5bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:42:57.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.225+0000 7f1d5bfff640 1 -- 172.21.15.57:0/1977639781 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d64146340 con 0x7f1d641457f0 2024-03-29T10:42:57.210 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.225+0000 7f1d5bfff640 10 monclient: _finish_auth 0 2024-03-29T10:42:57.210 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.225+0000 7f1d5bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:42:57.210 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.225+0000 7f1d5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:57.210 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.225+0000 7f1d5bfff640 1 -- 172.21.15.57:0/1977639781 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1d5c001690 con 0x7f1d641457f0 2024-03-29T10:42:57.210 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.225+0000 7f1d5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:57.210 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.225+0000 7f1d5bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:57.210 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.225+0000 7f1d5bfff640 10 monclient: dump: 2024-03-29T10:42:57.210 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:57.210 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:57.210 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:57.210 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:57.210 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:57.211 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:57.211 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:57.211 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:57.211 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:57.211 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:57.211 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.225+0000 7f1d6b69a640 5 monclient: authenticate success, global_id 4358 2024-03-29T10:42:57.211 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.225+0000 7f1d6b69a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:57.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.225+0000 7f1d5bfff640 1 -- 172.21.15.57:0/1977639781 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d5c0017f0 con 0x7f1d641457f0 2024-03-29T10:42:57.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.225+0000 7f1d5bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:57.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.225+0000 7f1d5bfff640 1 -- 172.21.15.57:0/1977639781 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1d5c001c00 con 0x7f1d641457f0 2024-03-29T10:42:57.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.226+0000 7f1d5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:57.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.226+0000 7f1d5bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:57.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.226+0000 7f1d5bfff640 10 monclient: dump: 2024-03-29T10:42:57.213 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:57.213 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:57.213 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:57.213 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:57.213 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:57.214 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:57.214 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:57.214 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:57.214 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:57.214 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:57.214 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.226+0000 7f1d6b69a640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:57.214 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.226+0000 7f1d6b69a640 4 set_mon_vals no callback set 2024-03-29T10:42:57.214 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.227+0000 7f1d6b69a640 10 monclient: shutdown 2024-03-29T10:42:57.214 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.227+0000 7f1d6b69a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:57.214 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.227+0000 7f1d6b69a640 1 -- 172.21.15.57:0/1977639781 >> v1:172.21.15.57:6790/0 conn(0x7f1d641457f0 legacy=0x7f1d6413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:57.214 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.227+0000 7f1d6b69a640 1 -- 172.21.15.57:0/1977639781 shutdown_connections 2024-03-29T10:42:57.214 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.227+0000 7f1d6b69a640 1 -- 172.21.15.57:0/1977639781 >> 172.21.15.57:0/1977639781 conn(0x7f1d64066a00 msgr2=0x7f1d64066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:57.215 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.228+0000 7f1d6b69a640 1 -- 172.21.15.57:0/1977639781 shutdown_connections 2024-03-29T10:42:57.215 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.228+0000 7f1d6b69a640 1 -- 172.21.15.57:0/1977639781 wait complete. 2024-03-29T10:42:57.215 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.229+0000 7f1d6b69a640 10 monclient: build_initial_monmap 2024-03-29T10:42:57.215 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.229+0000 7f1d6b69a640 10 monclient: monmap: 2024-03-29T10:42:57.215 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:42:57.215 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:57.215 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:42:57.215 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:42:57.215 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:57.215 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:57.215 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:42:57.215 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:42:57.215 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:42:57.216 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:57.219 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.229+0000 7f1d6b69a640 1 librados: starting msgr at 2024-03-29T10:42:57.220 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.229+0000 7f1d6b69a640 1 librados: starting objecter 2024-03-29T10:42:57.220 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.229+0000 7f1d6b69a640 1 Processor -- start 2024-03-29T10:42:57.220 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.230+0000 7f1d6b69a640 1 -- start start 2024-03-29T10:42:57.220 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.230+0000 7f1d6b69a640 1 librados: setting wanted keys 2024-03-29T10:42:57.220 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.230+0000 7f1d6b69a640 1 librados: calling monclient init 2024-03-29T10:42:57.220 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.230+0000 7f1d6b69a640 10 monclient: init 2024-03-29T10:42:57.220 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.230+0000 7f1d6b69a640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:57.220 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.230+0000 7f1d6b69a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:42:57.220 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.230+0000 7f1d6b69a640 10 monclient(hunting): picked mon.noname-b con 0x7f1d640a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T10:42:57.220 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d6b69a640 10 monclient(hunting): picked mon.noname-c con 0x7f1d6413f650 addr v1:172.21.15.57:6790/0 2024-03-29T10:42:57.220 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d69436640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f1d640a9d70 0x7f1d6413b7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:35654/0 (socket says 172.21.15.57:35654) 2024-03-29T10:42:57.220 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d69436640 1 -- 172.21.15.57:0/804421643 learned_addr learned my addr 172.21.15.57:0/804421643 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:42:57.221 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d6b69a640 10 monclient(hunting): picked mon.noname-a con 0x7f1d641457f0 addr v1:172.21.15.57:6789/0 2024-03-29T10:42:57.221 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d6b69a640 1 -- 172.21.15.57:0/804421643 --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d6406e050 con 0x7f1d641457f0 2024-03-29T10:42:57.221 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d6b69a640 1 -- 172.21.15.57:0/804421643 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d6406f480 con 0x7f1d640a9d70 2024-03-29T10:42:57.221 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d6b69a640 1 -- 172.21.15.57:0/804421643 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d6406f6f0 con 0x7f1d6413f650 2024-03-29T10:42:57.221 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d6b69a640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:57.221 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d6b69a640 10 monclient(hunting): authenticate will time out at 1129.407486s 2024-03-29T10:42:57.221 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d59ffb640 10 client.?.objecter ms_handle_connect 0x7f1d6413f650 2024-03-29T10:42:57.221 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d59ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:57.221 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d59ffb640 10 client.?.objecter ms_handle_connect 0x7f1d640a9d70 2024-03-29T10:42:57.221 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d59ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:57.221 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d59ffb640 10 client.?.objecter ms_handle_connect 0x7f1d641457f0 2024-03-29T10:42:57.221 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d59ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:57.221 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d59ffb640 1 -- 172.21.15.57:0/804421643 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 439234202 0 0) 0x7f1d6406f480 con 0x7f1d640a9d70 2024-03-29T10:42:57.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d59ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:42:57.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:57.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:57.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d59ffb640 10 monclient(hunting): my global_id is 4348 2024-03-29T10:42:57.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d59ffb640 1 -- 172.21.15.57:0/804421643 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d48004eb0 con 0x7f1d640a9d70 2024-03-29T10:42:57.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d59ffb640 1 -- 172.21.15.57:0/804421643 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989978997 0 0) 0x7f1d6406f6f0 con 0x7f1d6413f650 2024-03-29T10:42:57.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d59ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:42:57.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:57.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:57.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d59ffb640 10 monclient(hunting): my global_id is 4361 2024-03-29T10:42:57.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.231+0000 7f1d59ffb640 1 -- 172.21.15.57:0/804421643 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d6406f480 con 0x7f1d6413f650 2024-03-29T10:42:57.223 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 1 -- 172.21.15.57:0/804421643 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4066531906 0 0) 0x7f1d6406f480 con 0x7f1d6413f650 2024-03-29T10:42:57.223 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 1 -- 172.21.15.57:0/804421643 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d6406f6f0 con 0x7f1d6413f650 2024-03-29T10:42:57.223 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 1 -- 172.21.15.57:0/804421643 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1d60001cb0 con 0x7f1d6413f650 2024-03-29T10:42:57.223 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:57.223 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:57.223 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 10 monclient(hunting): dump: 2024-03-29T10:42:57.223 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:57.223 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:57.223 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:57.223 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:57.223 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:57.224 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:57.224 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:57.224 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:57.224 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:57.224 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:57.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:57.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 1 -- 172.21.15.57:0/804421643 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166992435 0 0) 0x7f1d6406e050 con 0x7f1d641457f0 2024-03-29T10:42:57.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:42:57.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:57.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:57.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 10 monclient(hunting): my global_id is 4335 2024-03-29T10:42:57.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 1 -- 172.21.15.57:0/804421643 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d6406f480 con 0x7f1d641457f0 2024-03-29T10:42:57.225 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 1 -- 172.21.15.57:0/804421643 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3006254533 0 0) 0x7f1d48004eb0 con 0x7f1d640a9d70 2024-03-29T10:42:57.225 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 1 -- 172.21.15.57:0/804421643 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d6406e050 con 0x7f1d640a9d70 2024-03-29T10:42:57.225 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 1 -- 172.21.15.57:0/804421643 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1d5c001b80 con 0x7f1d640a9d70 2024-03-29T10:42:57.225 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:57.225 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:57.225 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 10 monclient(hunting): dump: 2024-03-29T10:42:57.225 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:57.225 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:57.225 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:57.225 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:57.225 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:57.225 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:57.225 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:57.226 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:57.226 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:57.226 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:57.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 1 -- 172.21.15.57:0/804421643 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 866244957 0 0) 0x7f1d6406f6f0 con 0x7f1d6413f650 2024-03-29T10:42:57.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 1 -- 172.21.15.57:0/804421643 >> v1:172.21.15.179:6789/0 conn(0x7f1d640a9d70 legacy=0x7f1d6413b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:57.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 1 -- 172.21.15.57:0/804421643 >> v1:172.21.15.57:6789/0 conn(0x7f1d641457f0 legacy=0x7f1d641df080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:57.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:42:57.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 1 monclient: found mon.c 2024-03-29T10:42:57.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:57.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:42:57.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 1 -- 172.21.15.57:0/804421643 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d64138cc0 con 0x7f1d6413f650 2024-03-29T10:42:57.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 10 monclient: _finish_auth 0 2024-03-29T10:42:57.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:42:57.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:57.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d6b69a640 5 monclient: authenticate success, global_id 4361 2024-03-29T10:42:57.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d6b69a640 10 monclient: _renew_subs 2024-03-29T10:42:57.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d6b69a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:42:57.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.232+0000 7f1d6b69a640 1 -- 172.21.15.57:0/804421643 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d64138f10 con 0x7f1d6413f650 2024-03-29T10:42:57.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.233+0000 7f1d6b69a640 10 client.4361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:57.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.233+0000 7f1d6b69a640 10 monclient: _renew_subs 2024-03-29T10:42:57.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.233+0000 7f1d6b69a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:42:57.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.233+0000 7f1d6b69a640 1 -- 172.21.15.57:0/804421643 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d64138190 con 0x7f1d6413f650 2024-03-29T10:42:57.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.233+0000 7f1d6b69a640 1 librados: init done 2024-03-29T10:42:57.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.233+0000 7f1d59ffb640 1 -- 172.21.15.57:0/804421643 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d60003990 con 0x7f1d6413f650 2024-03-29T10:42:57.228 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.233+0000 7f1d59ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:57.228 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.233+0000 7f1d59ffb640 1 -- 172.21.15.57:0/804421643 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1d60003ce0 con 0x7f1d6413f650 2024-03-29T10:42:57.228 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.233+0000 7f1d59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:57.228 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.233+0000 7f1d59ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:57.228 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.233+0000 7f1d59ffb640 10 monclient: dump: 2024-03-29T10:42:57.228 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:42:57.228 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:42:57.228 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:57.228 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:42:57.228 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:57.228 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:42:57.228 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:42:57.228 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:42:57.229 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:42:57.229 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:42:57.229 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.235+0000 7f1d59ffb640 1 -- 172.21.15.57:0/804421643 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f1d60003e40 con 0x7f1d6413f650 2024-03-29T10:42:57.229 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.236+0000 7f1d59ffb640 1 -- 172.21.15.57:0/804421643 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f1d60083b70 con 0x7f1d6413f650 2024-03-29T10:42:57.229 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.236+0000 7f1d59ffb640 10 client.4361.objecter ms_dispatch 0x7f1d64000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:57.229 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.236+0000 7f1d59ffb640 3 client.4361.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:57.229 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.236+0000 7f1d59ffb640 3 client.4361.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:57.229 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.236+0000 7f1d59ffb640 20 client.4361.objecter dump_active .. 0 homeless 2024-03-29T10:42:57.230 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.236+0000 7f1d5b7fe640 4 set_mon_vals no callback set 2024-03-29T10:42:57.230 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.236+0000 7f1d69436640 10 monclient: get_auth_request con 0x7f1d4806e000 auth_method 0 2024-03-29T10:42:57.230 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.236+0000 7f1d59ffb640 10 client.4361.objecter ms_handle_connect 0x7f1d4806e000 2024-03-29T10:42:57.230 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.237+0000 7f1d6b69a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:57.230 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.237+0000 7f1d6b69a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:57.230 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.237+0000 7f1d6b69a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:42:57.230 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.237+0000 7f1d6b69a640 1 -- 172.21.15.57:0/804421643 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d44005080 con 0x7f1d6413f650 2024-03-29T10:42:57.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.249+0000 7f1d59ffb640 1 -- 172.21.15.57:0/804421643 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1d6004f450 con 0x7f1d6413f650 2024-03-29T10:42:57.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.249+0000 7f1d59ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:57.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.249+0000 7f1d59ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:57.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.616+0000 7f1d6b69a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:57.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.616+0000 7f1d6b69a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:57.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.616+0000 7f1d6b69a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:42:57.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.616+0000 7f1d6b69a640 1 -- 172.21.15.57:0/804421643 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d44005330 con 0x7f1d6413f650 2024-03-29T10:42:57.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.617+0000 7f1d59ffb640 1 -- 172.21.15.57:0/804421643 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f1d60051380 con 0x7f1d6413f650 2024-03-29T10:42:57.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.617+0000 7f1d59ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:57.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.617+0000 7f1d59ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:57.600 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:42:57.600 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:57.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.620+0000 7f1d6b69a640 10 librados: watch_flush enter 2024-03-29T10:42:57.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.620+0000 7f1d6b69a640 10 librados: watch_flush exit 2024-03-29T10:42:57.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.620+0000 7f1d6b69a640 20 client.4361.objecter shutdown clearing up homeless session... 2024-03-29T10:42:57.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.620+0000 7f1d6b69a640 10 client.4361.objecter successfully canceled tick 2024-03-29T10:42:57.604 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.621+0000 7f1d6b69a640 1 -- 172.21.15.57:0/804421643 >> v1:172.21.15.179:6816/2698295936 conn(0x7f1d4806e000 legacy=0x7f1d480704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:57.604 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.621+0000 7f1d6b69a640 10 monclient: shutdown 2024-03-29T10:42:57.604 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.621+0000 7f1d6b69a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:57.604 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.621+0000 7f1d6b69a640 1 -- 172.21.15.57:0/804421643 >> v1:172.21.15.57:6790/0 conn(0x7f1d6413f650 legacy=0x7f1d641378a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:57.604 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.621+0000 7f1d6b69a640 1 -- 172.21.15.57:0/804421643 shutdown_connections 2024-03-29T10:42:57.604 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.621+0000 7f1d6b69a640 1 -- 172.21.15.57:0/804421643 >> 172.21.15.57:0/804421643 conn(0x7f1d64066a00 msgr2=0x7f1d640b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:57.604 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.621+0000 7f1d6b69a640 1 -- 172.21.15.57:0/804421643 shutdown_connections 2024-03-29T10:42:57.605 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.621+0000 7f1d6b69a640 1 -- 172.21.15.57:0/804421643 wait complete. 2024-03-29T10:42:57.605 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:42:57.621+0000 7f1d6b69a640 1 librados: shutdown 2024-03-29T10:42:57.619 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:04.426 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:04.578 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.595+0000 7fe23e614640 10 monclient: get_monmap_and_config 2024-03-29T10:43:04.579 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.595+0000 7fe23e614640 10 monclient: build_initial_monmap 2024-03-29T10:43:04.579 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.595+0000 7fe23e614640 10 monclient: monmap: 2024-03-29T10:43:04.579 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:43:04.579 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:04.579 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:43:04.596276+0000 2024-03-29T10:43:04.579 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:43:04.596276+0000 2024-03-29T10:43:04.579 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:04.579 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:04.579 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:43:04.579 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:43:04.579 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:43:04.579 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:04.582 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.596+0000 7fe23e614640 1 Processor -- start 2024-03-29T10:43:04.582 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.596+0000 7fe23e614640 1 -- start start 2024-03-29T10:43:04.582 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.596+0000 7fe23e614640 10 monclient: init 2024-03-29T10:43:04.582 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.597+0000 7fe23e614640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:04.582 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.597+0000 7fe23e614640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:43:04.582 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.597+0000 7fe23e614640 10 monclient(hunting): picked mon.noname-b con 0x7fe2380b0770 addr v1:172.21.15.57:6790/0 2024-03-29T10:43:04.582 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.597+0000 7fe23e614640 10 monclient(hunting): picked mon.noname-c con 0x7fe23814a700 addr v1:172.21.15.179:6789/0 2024-03-29T10:43:04.582 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.597+0000 7fe23e614640 10 monclient(hunting): picked mon.noname-a con 0x7fe2380ac5a0 addr v1:172.21.15.57:6789/0 2024-03-29T10:43:04.582 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.597+0000 7fe23e614640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2380aba70 con 0x7fe2380ac5a0 2024-03-29T10:43:04.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.597+0000 7fe23e614640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2380aecb0 con 0x7fe23814a700 2024-03-29T10:43:04.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.597+0000 7fe23e614640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2380af150 con 0x7fe2380b0770 2024-03-29T10:43:04.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.597+0000 7fe23e614640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:04.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.597+0000 7fe23e614640 10 monclient(hunting): authenticate will time out at 1136.773861s 2024-03-29T10:43:04.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.597+0000 7fe2377fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fe23814a700 0x7fe2380abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:58648/0 (socket says 172.21.15.57:58648) 2024-03-29T10:43:04.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.597+0000 7fe2377fe640 1 -- 172.21.15.57:0/3542553482 learned_addr learned my addr 172.21.15.57:0/3542553482 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:43:04.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.598+0000 7fe236ffd640 1 -- 172.21.15.57:0/3542553482 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 749009474 0 0) 0x7fe2380aecb0 con 0x7fe23814a700 2024-03-29T10:43:04.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.598+0000 7fe236ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:04.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.598+0000 7fe236ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:04.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.598+0000 7fe236ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:04.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.598+0000 7fe236ffd640 10 monclient(hunting): my global_id is 4351 2024-03-29T10:43:04.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.598+0000 7fe236ffd640 1 -- 172.21.15.57:0/3542553482 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe220003520 con 0x7fe23814a700 2024-03-29T10:43:04.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.598+0000 7fe236ffd640 1 -- 172.21.15.57:0/3542553482 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738607569 0 0) 0x7fe2380aba70 con 0x7fe2380ac5a0 2024-03-29T10:43:04.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.599+0000 7fe236ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:04.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.599+0000 7fe236ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:04.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.599+0000 7fe236ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:04.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.599+0000 7fe236ffd640 10 monclient(hunting): my global_id is 4338 2024-03-29T10:43:04.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.599+0000 7fe236ffd640 1 -- 172.21.15.57:0/3542553482 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2380aecb0 con 0x7fe2380ac5a0 2024-03-29T10:43:04.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.599+0000 7fe236ffd640 1 -- 172.21.15.57:0/3542553482 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3258674963 0 0) 0x7fe220003520 con 0x7fe23814a700 2024-03-29T10:43:04.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.599+0000 7fe236ffd640 1 -- 172.21.15.57:0/3542553482 >> v1:172.21.15.57:6790/0 conn(0x7fe2380b0770 legacy=0x7fe238149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:04.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.599+0000 7fe236ffd640 1 -- 172.21.15.57:0/3542553482 >> v1:172.21.15.57:6789/0 conn(0x7fe2380ac5a0 legacy=0x7fe238151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:04.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.599+0000 7fe236ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:43:04.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.599+0000 7fe236ffd640 1 monclient: found mon.noname-c 2024-03-29T10:43:04.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.599+0000 7fe236ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:04.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.599+0000 7fe236ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:43:04.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.599+0000 7fe236ffd640 1 -- 172.21.15.57:0/3542553482 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2380ac9d0 con 0x7fe23814a700 2024-03-29T10:43:04.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.599+0000 7fe236ffd640 10 monclient: _finish_auth 0 2024-03-29T10:43:04.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.599+0000 7fe236ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:43:04.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.599+0000 7fe236ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:04.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.599+0000 7fe236ffd640 1 -- 172.21.15.57:0/3542553482 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fe2280016b0 con 0x7fe23814a700 2024-03-29T10:43:04.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.599+0000 7fe236ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:04.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.599+0000 7fe236ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:04.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.599+0000 7fe236ffd640 10 monclient: dump: 2024-03-29T10:43:04.585 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:04.585 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:04.585 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:04.585 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:04.585 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:04.585 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:04.585 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:04.585 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:04.585 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:04.585 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:04.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.599+0000 7fe23e614640 5 monclient: authenticate success, global_id 4351 2024-03-29T10:43:04.586 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.599+0000 7fe23e614640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:04.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.600+0000 7fe236ffd640 1 -- 172.21.15.57:0/3542553482 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe228001810 con 0x7fe23814a700 2024-03-29T10:43:04.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.600+0000 7fe236ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:04.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.600+0000 7fe236ffd640 1 -- 172.21.15.57:0/3542553482 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fe228001be0 con 0x7fe23814a700 2024-03-29T10:43:04.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.600+0000 7fe236ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:04.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.600+0000 7fe236ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:04.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.600+0000 7fe236ffd640 10 monclient: dump: 2024-03-29T10:43:04.588 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:04.588 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:04.588 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:04.588 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:04.588 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:04.588 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:04.588 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:04.589 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:04.589 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:04.589 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:04.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.600+0000 7fe23e614640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:04.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.600+0000 7fe23e614640 4 set_mon_vals no callback set 2024-03-29T10:43:04.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.601+0000 7fe23e614640 10 monclient: shutdown 2024-03-29T10:43:04.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.601+0000 7fe23e614640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:04.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.601+0000 7fe23e614640 1 -- 172.21.15.57:0/3542553482 >> v1:172.21.15.179:6789/0 conn(0x7fe23814a700 legacy=0x7fe2380abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:04.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.601+0000 7fe23e614640 1 -- 172.21.15.57:0/3542553482 shutdown_connections 2024-03-29T10:43:04.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.601+0000 7fe23e614640 1 -- 172.21.15.57:0/3542553482 >> 172.21.15.57:0/3542553482 conn(0x7fe23806b660 msgr2=0x7fe23806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:04.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.602+0000 7fe23e614640 1 -- 172.21.15.57:0/3542553482 shutdown_connections 2024-03-29T10:43:04.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.602+0000 7fe23e614640 1 -- 172.21.15.57:0/3542553482 wait complete. 2024-03-29T10:43:04.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.603+0000 7fe23e614640 10 monclient: build_initial_monmap 2024-03-29T10:43:04.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.603+0000 7fe23e614640 10 monclient: monmap: 2024-03-29T10:43:04.589 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:43:04.590 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:04.590 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:43:04.590 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:43:04.590 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:04.590 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:04.590 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:43:04.590 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:43:04.590 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:43:04.590 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:04.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.604+0000 7fe23e614640 1 librados: starting msgr at 2024-03-29T10:43:04.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.604+0000 7fe23e614640 1 librados: starting objecter 2024-03-29T10:43:04.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.604+0000 7fe23e614640 1 Processor -- start 2024-03-29T10:43:04.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.604+0000 7fe23e614640 1 -- start start 2024-03-29T10:43:04.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.604+0000 7fe23e614640 1 librados: setting wanted keys 2024-03-29T10:43:04.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.604+0000 7fe23e614640 1 librados: calling monclient init 2024-03-29T10:43:04.591 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.604+0000 7fe23e614640 10 monclient: init 2024-03-29T10:43:04.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.605+0000 7fe23e614640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:04.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.605+0000 7fe23e614640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:43:04.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.605+0000 7fe23e614640 10 monclient(hunting): picked mon.noname-b con 0x7fe2380ac5a0 addr v1:172.21.15.179:6789/0 2024-03-29T10:43:04.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.605+0000 7fe23e614640 10 monclient(hunting): picked mon.noname-c con 0x7fe2380b0770 addr v1:172.21.15.57:6790/0 2024-03-29T10:43:04.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.605+0000 7fe23e614640 10 monclient(hunting): picked mon.noname-a con 0x7fe23814a700 addr v1:172.21.15.57:6789/0 2024-03-29T10:43:04.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.605+0000 7fe23e614640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe238126630 con 0x7fe23814a700 2024-03-29T10:43:04.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.605+0000 7fe23e614640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe238128290 con 0x7fe2380ac5a0 2024-03-29T10:43:04.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.605+0000 7fe23e614640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe238125f10 con 0x7fe2380b0770 2024-03-29T10:43:04.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.605+0000 7fe23e614640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:04.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.605+0000 7fe23e614640 10 monclient(hunting): authenticate will time out at 1136.782022s 2024-03-29T10:43:04.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.606+0000 7fe23cbb1640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fe23814a700 0x7fe2381df380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:35550/0 (socket says 172.21.15.57:35550) 2024-03-29T10:43:04.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.606+0000 7fe23cbb1640 1 -- 172.21.15.57:0/2394238691 learned_addr learned my addr 172.21.15.57:0/2394238691 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:43:04.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.606+0000 7fe234ff9640 10 client.?.objecter ms_handle_connect 0x7fe2380b0770 2024-03-29T10:43:04.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.606+0000 7fe234ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:04.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.606+0000 7fe234ff9640 10 client.?.objecter ms_handle_connect 0x7fe23814a700 2024-03-29T10:43:04.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.606+0000 7fe234ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:04.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.607+0000 7fe234ff9640 1 -- 172.21.15.57:0/2394238691 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378289105 0 0) 0x7fe238126630 con 0x7fe23814a700 2024-03-29T10:43:04.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.607+0000 7fe234ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:43:04.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.607+0000 7fe234ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:04.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.607+0000 7fe234ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:04.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.607+0000 7fe234ff9640 10 monclient(hunting): my global_id is 4341 2024-03-29T10:43:04.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.607+0000 7fe234ff9640 1 -- 172.21.15.57:0/2394238691 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe224004eb0 con 0x7fe23814a700 2024-03-29T10:43:04.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.607+0000 7fe234ff9640 1 -- 172.21.15.57:0/2394238691 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028523669 0 0) 0x7fe238125f10 con 0x7fe2380b0770 2024-03-29T10:43:04.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.607+0000 7fe234ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:43:04.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.607+0000 7fe234ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:04.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.607+0000 7fe234ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:04.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.607+0000 7fe234ff9640 10 monclient(hunting): my global_id is 4367 2024-03-29T10:43:04.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.607+0000 7fe234ff9640 1 -- 172.21.15.57:0/2394238691 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe238126630 con 0x7fe2380b0770 2024-03-29T10:43:04.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.607+0000 7fe234ff9640 10 client.?.objecter ms_handle_connect 0x7fe2380ac5a0 2024-03-29T10:43:04.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.607+0000 7fe234ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:04.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.608+0000 7fe234ff9640 1 -- 172.21.15.57:0/2394238691 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3578922307 0 0) 0x7fe224004eb0 con 0x7fe23814a700 2024-03-29T10:43:04.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.608+0000 7fe234ff9640 1 -- 172.21.15.57:0/2394238691 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe238125f10 con 0x7fe23814a700 2024-03-29T10:43:04.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.608+0000 7fe234ff9640 1 -- 172.21.15.57:0/2394238691 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fe230001d10 con 0x7fe23814a700 2024-03-29T10:43:04.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.608+0000 7fe234ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:04.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.608+0000 7fe234ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:04.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.608+0000 7fe234ff9640 10 monclient(hunting): dump: 2024-03-29T10:43:04.597 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:04.597 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:04.597 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:04.597 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:04.597 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:04.597 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:04.597 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:04.597 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:04.598 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:04.598 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:04.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.608+0000 7fe234ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:04.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.608+0000 7fe234ff9640 1 -- 172.21.15.57:0/2394238691 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806179383 0 0) 0x7fe238128290 con 0x7fe2380ac5a0 2024-03-29T10:43:04.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.608+0000 7fe234ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:43:04.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.608+0000 7fe234ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:04.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.608+0000 7fe234ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:04.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.608+0000 7fe234ff9640 10 monclient(hunting): my global_id is 4354 2024-03-29T10:43:04.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.608+0000 7fe234ff9640 1 -- 172.21.15.57:0/2394238691 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe224004eb0 con 0x7fe2380ac5a0 2024-03-29T10:43:04.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.608+0000 7fe234ff9640 1 -- 172.21.15.57:0/2394238691 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1070758777 0 0) 0x7fe238126630 con 0x7fe2380b0770 2024-03-29T10:43:04.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.608+0000 7fe234ff9640 1 -- 172.21.15.57:0/2394238691 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe238128290 con 0x7fe2380b0770 2024-03-29T10:43:04.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.608+0000 7fe234ff9640 1 -- 172.21.15.57:0/2394238691 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fe228001b20 con 0x7fe2380b0770 2024-03-29T10:43:04.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.608+0000 7fe234ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:04.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.608+0000 7fe234ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:04.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.608+0000 7fe234ff9640 10 monclient(hunting): dump: 2024-03-29T10:43:04.599 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:04.599 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:04.599 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:04.599 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:04.599 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:04.599 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:04.599 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:04.599 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:04.599 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:04.599 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:04.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.608+0000 7fe234ff9640 1 -- 172.21.15.57:0/2394238691 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 798791424 0 0) 0x7fe238125f10 con 0x7fe23814a700 2024-03-29T10:43:04.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.608+0000 7fe234ff9640 1 -- 172.21.15.57:0/2394238691 >> v1:172.21.15.57:6790/0 conn(0x7fe2380b0770 legacy=0x7fe2381dec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:04.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.608+0000 7fe234ff9640 1 -- 172.21.15.57:0/2394238691 >> v1:172.21.15.179:6789/0 conn(0x7fe2380ac5a0 legacy=0x7fe238144e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:04.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.609+0000 7fe234ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:43:04.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.609+0000 7fe234ff9640 1 monclient: found mon.a 2024-03-29T10:43:04.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.609+0000 7fe234ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:04.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.609+0000 7fe234ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:04.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.609+0000 7fe234ff9640 1 -- 172.21.15.57:0/2394238691 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe238145b90 con 0x7fe23814a700 2024-03-29T10:43:04.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.609+0000 7fe234ff9640 10 monclient: _finish_auth 0 2024-03-29T10:43:04.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.609+0000 7fe234ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:43:04.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.609+0000 7fe234ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:04.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.609+0000 7fe23e614640 5 monclient: authenticate success, global_id 4341 2024-03-29T10:43:04.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.609+0000 7fe23e614640 10 monclient: _renew_subs 2024-03-29T10:43:04.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.609+0000 7fe23e614640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:04.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.609+0000 7fe23e614640 1 -- 172.21.15.57:0/2394238691 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2381261c0 con 0x7fe23814a700 2024-03-29T10:43:04.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.609+0000 7fe23e614640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:04.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.609+0000 7fe23e614640 10 monclient: _renew_subs 2024-03-29T10:43:04.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.609+0000 7fe23e614640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:04.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.609+0000 7fe23e614640 1 -- 172.21.15.57:0/2394238691 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe238126d60 con 0x7fe23814a700 2024-03-29T10:43:04.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.609+0000 7fe23e614640 1 librados: init done 2024-03-29T10:43:04.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.610+0000 7fe234ff9640 1 -- 172.21.15.57:0/2394238691 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe230001e70 con 0x7fe23814a700 2024-03-29T10:43:04.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.610+0000 7fe234ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:04.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.610+0000 7fe234ff9640 1 -- 172.21.15.57:0/2394238691 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fe230003de0 con 0x7fe23814a700 2024-03-29T10:43:04.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.610+0000 7fe234ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:04.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.610+0000 7fe234ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:04.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.610+0000 7fe2367fc640 4 set_mon_vals no callback set 2024-03-29T10:43:04.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.610+0000 7fe234ff9640 10 monclient: dump: 2024-03-29T10:43:04.601 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:04.601 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:04.601 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:04.601 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:04.601 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:04.601 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:04.601 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:04.601 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:04.602 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:04.602 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:04.602 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.611+0000 7fe23e614640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:04.602 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.611+0000 7fe23e614640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:04.602 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.611+0000 7fe23e614640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:04.602 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.611+0000 7fe23e614640 1 -- 172.21.15.57:0/2394238691 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe218005080 con 0x7fe23814a700 2024-03-29T10:43:04.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.612+0000 7fe234ff9640 1 -- 172.21.15.57:0/2394238691 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fe230003f40 con 0x7fe23814a700 2024-03-29T10:43:04.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.613+0000 7fe234ff9640 1 -- 172.21.15.57:0/2394238691 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7fe230084df0 con 0x7fe23814a700 2024-03-29T10:43:04.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.613+0000 7fe234ff9640 10 client.4341.objecter ms_dispatch 0x7fe238000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:04.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.613+0000 7fe234ff9640 3 client.4341.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:04.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.613+0000 7fe234ff9640 3 client.4341.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:04.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.613+0000 7fe237fff640 10 monclient: get_auth_request con 0x7fe22406e000 auth_method 0 2024-03-29T10:43:04.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.613+0000 7fe234ff9640 20 client.4341.objecter dump_active .. 0 homeless 2024-03-29T10:43:04.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.614+0000 7fe234ff9640 10 client.4341.objecter ms_handle_connect 0x7fe22406e000 2024-03-29T10:43:04.609 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.626+0000 7fe234ff9640 1 -- 172.21.15.57:0/2394238691 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe2300506d0 con 0x7fe23814a700 2024-03-29T10:43:04.609 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.626+0000 7fe234ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:04.609 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:04.626+0000 7fe234ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:05.001 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:05.018+0000 7fe23e614640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:05.001 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:05.018+0000 7fe23e614640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:05.001 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:05.018+0000 7fe23e614640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:05.002 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:05.018+0000 7fe23e614640 1 -- 172.21.15.57:0/2394238691 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe218005330 con 0x7fe23814a700 2024-03-29T10:43:05.002 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:05.019+0000 7fe234ff9640 1 -- 172.21.15.57:0/2394238691 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7fe230052600 con 0x7fe23814a700 2024-03-29T10:43:05.002 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:05.019+0000 7fe234ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:05.002 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:05.019+0000 7fe234ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:05.003 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:43:05.003 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:05.007 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:05.023+0000 7fe23e614640 10 librados: watch_flush enter 2024-03-29T10:43:05.007 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:05.023+0000 7fe23e614640 10 librados: watch_flush exit 2024-03-29T10:43:05.007 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:05.023+0000 7fe23e614640 20 client.4341.objecter shutdown clearing up homeless session... 2024-03-29T10:43:05.007 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:05.023+0000 7fe23e614640 10 client.4341.objecter successfully canceled tick 2024-03-29T10:43:05.007 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:05.023+0000 7fe23e614640 1 -- 172.21.15.57:0/2394238691 >> v1:172.21.15.179:6816/2698295936 conn(0x7fe22406e000 legacy=0x7fe2240704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:05.007 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:05.023+0000 7fe23e614640 10 monclient: shutdown 2024-03-29T10:43:05.007 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:05.023+0000 7fe23e614640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:05.007 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:05.023+0000 7fe23e614640 1 -- 172.21.15.57:0/2394238691 >> v1:172.21.15.57:6789/0 conn(0x7fe23814a700 legacy=0x7fe2381df380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:05.007 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:05.024+0000 7fe23e614640 1 -- 172.21.15.57:0/2394238691 shutdown_connections 2024-03-29T10:43:05.007 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:05.024+0000 7fe23e614640 1 -- 172.21.15.57:0/2394238691 >> 172.21.15.57:0/2394238691 conn(0x7fe23806b660 msgr2=0x7fe2380b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:05.007 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:05.024+0000 7fe23e614640 1 -- 172.21.15.57:0/2394238691 shutdown_connections 2024-03-29T10:43:05.007 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:05.024+0000 7fe23e614640 1 -- 172.21.15.57:0/2394238691 wait complete. 2024-03-29T10:43:05.007 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:05.024+0000 7fe23e614640 1 librados: shutdown 2024-03-29T10:43:05.021 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:11.829 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:11.983 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:11.999+0000 7f512fb72640 10 monclient: get_monmap_and_config 2024-03-29T10:43:11.983 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:11.999+0000 7f512fb72640 10 monclient: build_initial_monmap 2024-03-29T10:43:11.983 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:11.999+0000 7f512fb72640 10 monclient: monmap: 2024-03-29T10:43:11.983 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:43:11.983 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:11.983 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:43:12.000779+0000 2024-03-29T10:43:11.983 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:43:12.000779+0000 2024-03-29T10:43:11.983 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:11.983 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:11.984 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:43:11.984 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:43:11.984 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:43:11.984 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:11.984 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.000+0000 7f512fb72640 1 Processor -- start 2024-03-29T10:43:11.987 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.001+0000 7f512fb72640 1 -- start start 2024-03-29T10:43:11.987 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.001+0000 7f512fb72640 10 monclient: init 2024-03-29T10:43:11.987 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.001+0000 7f512fb72640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:11.987 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.001+0000 7f512fb72640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:43:11.987 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.002+0000 7f512fb72640 10 monclient(hunting): picked mon.noname-c con 0x7f51281457f0 addr v1:172.21.15.179:6789/0 2024-03-29T10:43:11.988 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.002+0000 7f512fb72640 10 monclient(hunting): picked mon.noname-b con 0x7f512813f650 addr v1:172.21.15.57:6790/0 2024-03-29T10:43:11.988 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.002+0000 7f512fb72640 10 monclient(hunting): picked mon.noname-a con 0x7f51280a9d70 addr v1:172.21.15.57:6789/0 2024-03-29T10:43:11.988 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.002+0000 7f512fb72640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51280aaef0 con 0x7f51280a9d70 2024-03-29T10:43:11.988 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.002+0000 7f512fb72640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5128145bd0 con 0x7f51281457f0 2024-03-29T10:43:11.988 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.002+0000 7f512fb72640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5128145eb0 con 0x7f512813f650 2024-03-29T10:43:11.988 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.002+0000 7f512fb72640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:11.988 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.002+0000 7f512fb72640 10 monclient(hunting): authenticate will time out at 1144.179110s 2024-03-29T10:43:11.989 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.005+0000 7f512e10f640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f51280a9d70 0x7f512814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:34321/0 (socket says 172.21.15.57:34321) 2024-03-29T10:43:11.989 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.005+0000 7f512e10f640 1 -- 172.21.15.57:0/4115868590 learned_addr learned my addr 172.21.15.57:0/4115868590 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:43:11.990 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.005+0000 7f512c90c640 1 -- 172.21.15.57:0/4115868590 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2444199592 0 0) 0x7f51280aaef0 con 0x7f51280a9d70 2024-03-29T10:43:11.990 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.005+0000 7f512c90c640 10 monclient(hunting): _negotiate 2024-03-29T10:43:11.990 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.005+0000 7f512c90c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:11.990 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.005+0000 7f512c90c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:11.990 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.005+0000 7f512c90c640 10 monclient(hunting): my global_id is 4344 2024-03-29T10:43:11.990 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.005+0000 7f512c90c640 1 -- 172.21.15.57:0/4115868590 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5114003520 con 0x7f51280a9d70 2024-03-29T10:43:11.991 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.006+0000 7f512c90c640 1 -- 172.21.15.57:0/4115868590 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4075004878 0 0) 0x7f5114003520 con 0x7f51280a9d70 2024-03-29T10:43:11.991 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.006+0000 7f512c90c640 1 -- 172.21.15.57:0/4115868590 >> v1:172.21.15.57:6790/0 conn(0x7f512813f650 legacy=0x7f51280b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:43:11.991 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.007+0000 7f512c90c640 1 -- 172.21.15.57:0/4115868590 >> v1:172.21.15.179:6789/0 conn(0x7f51281457f0 legacy=0x7f512813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:11.992 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.007+0000 7f512c90c640 10 monclient: _finish_hunting 0 2024-03-29T10:43:11.992 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.007+0000 7f512c90c640 1 monclient: found mon.noname-a 2024-03-29T10:43:11.992 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.007+0000 7f512c90c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:11.992 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.007+0000 7f512c90c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:43:11.992 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.007+0000 7f512c90c640 1 -- 172.21.15.57:0/4115868590 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5128146340 con 0x7f51280a9d70 2024-03-29T10:43:11.993 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.007+0000 7f512c90c640 10 monclient: _finish_auth 0 2024-03-29T10:43:11.993 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.007+0000 7f512c90c640 10 monclient: _check_auth_tickets 2024-03-29T10:43:11.993 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.007+0000 7f512c90c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:11.993 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.007+0000 7f512c90c640 1 -- 172.21.15.57:0/4115868590 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f5120001a10 con 0x7f51280a9d70 2024-03-29T10:43:11.993 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.007+0000 7f512c90c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:11.993 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.007+0000 7f512c90c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:11.994 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.007+0000 7f512c90c640 10 monclient: dump: 2024-03-29T10:43:11.994 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:11.994 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:11.994 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:11.994 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:11.994 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:11.994 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:11.994 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:11.994 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:11.994 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:11.994 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:11.994 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.008+0000 7f512fb72640 5 monclient: authenticate success, global_id 4344 2024-03-29T10:43:11.995 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.008+0000 7f512fb72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:11.995 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.008+0000 7f512c90c640 1 -- 172.21.15.57:0/4115868590 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5120001be0 con 0x7f51280a9d70 2024-03-29T10:43:11.995 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.008+0000 7f512c90c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:11.995 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.008+0000 7f512fb72640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:11.995 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.008+0000 7f512fb72640 4 set_mon_vals no callback set 2024-03-29T10:43:11.996 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.009+0000 7f512fb72640 10 monclient: shutdown 2024-03-29T10:43:11.996 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.009+0000 7f512fb72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:11.996 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.009+0000 7f512fb72640 1 -- 172.21.15.57:0/4115868590 >> v1:172.21.15.57:6789/0 conn(0x7f51280a9d70 legacy=0x7f512814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:11.996 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.009+0000 7f512fb72640 1 -- 172.21.15.57:0/4115868590 shutdown_connections 2024-03-29T10:43:11.996 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.009+0000 7f512fb72640 1 -- 172.21.15.57:0/4115868590 >> 172.21.15.57:0/4115868590 conn(0x7f5128066a00 msgr2=0x7f5128066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:11.996 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.011+0000 7f512fb72640 1 -- 172.21.15.57:0/4115868590 shutdown_connections 2024-03-29T10:43:11.996 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.011+0000 7f512fb72640 1 -- 172.21.15.57:0/4115868590 wait complete. 2024-03-29T10:43:11.997 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.012+0000 7f512fb72640 10 monclient: build_initial_monmap 2024-03-29T10:43:11.998 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.012+0000 7f512fb72640 10 monclient: monmap: 2024-03-29T10:43:11.998 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:43:11.998 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:11.998 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:43:11.998 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:43:11.998 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:11.998 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:11.998 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:43:11.998 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:43:11.998 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:43:11.998 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:11.998 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.012+0000 7f512fb72640 1 librados: starting msgr at 2024-03-29T10:43:11.998 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.012+0000 7f512fb72640 1 librados: starting objecter 2024-03-29T10:43:11.999 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.012+0000 7f512fb72640 1 Processor -- start 2024-03-29T10:43:11.999 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.012+0000 7f512fb72640 1 -- start start 2024-03-29T10:43:11.999 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.012+0000 7f512fb72640 1 librados: setting wanted keys 2024-03-29T10:43:11.999 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.012+0000 7f512fb72640 1 librados: calling monclient init 2024-03-29T10:43:11.999 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.012+0000 7f512fb72640 10 monclient: init 2024-03-29T10:43:11.999 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.013+0000 7f512fb72640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:11.999 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.013+0000 7f512fb72640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:43:11.999 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.013+0000 7f512fb72640 10 monclient(hunting): picked mon.noname-a con 0x7f51280a9d70 addr v1:172.21.15.57:6789/0 2024-03-29T10:43:12.000 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.017+0000 7f512fb72640 10 monclient(hunting): picked mon.noname-c con 0x7f512813f650 addr v1:172.21.15.57:6790/0 2024-03-29T10:43:12.009 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.017+0000 7f512fb72640 10 monclient(hunting): picked mon.noname-b con 0x7f51281457f0 addr v1:172.21.15.179:6789/0 2024-03-29T10:43:12.009 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.017+0000 7f512fb72640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5128070e10 con 0x7f51280a9d70 2024-03-29T10:43:12.009 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.017+0000 7f512fb72640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5128071fc0 con 0x7f51281457f0 2024-03-29T10:43:12.009 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.017+0000 7f512fb72640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51280708f0 con 0x7f512813f650 2024-03-29T10:43:12.009 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.017+0000 7f512fb72640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:12.009 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.017+0000 7f512fb72640 10 monclient(hunting): authenticate will time out at 1144.194692s 2024-03-29T10:43:12.009 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.017+0000 7f512d90e640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f51280a9d70 0x7f51280b0da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:34479/0 (socket says 172.21.15.57:34479) 2024-03-29T10:43:12.009 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.017+0000 7f512d90e640 1 -- 172.21.15.57:0/3849474291 learned_addr learned my addr 172.21.15.57:0/3849474291 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:43:12.009 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.018+0000 7f511e7fc640 10 client.?.objecter ms_handle_connect 0x7f512813f650 2024-03-29T10:43:12.009 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.018+0000 7f511e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:12.009 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.018+0000 7f511e7fc640 10 client.?.objecter ms_handle_connect 0x7f51280a9d70 2024-03-29T10:43:12.009 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.018+0000 7f511e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:12.009 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.018+0000 7f511e7fc640 1 -- 172.21.15.57:0/3849474291 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1598567485 0 0) 0x7f5128070e10 con 0x7f51280a9d70 2024-03-29T10:43:12.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.018+0000 7f511e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:12.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.018+0000 7f511e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:12.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.018+0000 7f511e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:12.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.018+0000 7f511e7fc640 10 monclient(hunting): my global_id is 4347 2024-03-29T10:43:12.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.018+0000 7f511e7fc640 1 -- 172.21.15.57:0/3849474291 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f510c004eb0 con 0x7f51280a9d70 2024-03-29T10:43:12.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.019+0000 7f511e7fc640 1 -- 172.21.15.57:0/3849474291 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 798777557 0 0) 0x7f510c004eb0 con 0x7f51280a9d70 2024-03-29T10:43:12.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.019+0000 7f511e7fc640 1 -- 172.21.15.57:0/3849474291 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5128070e10 con 0x7f51280a9d70 2024-03-29T10:43:12.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.019+0000 7f511e7fc640 1 -- 172.21.15.57:0/3849474291 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f5124003f10 con 0x7f51280a9d70 2024-03-29T10:43:12.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.019+0000 7f511e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:12.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.019+0000 7f511e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:12.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.019+0000 7f511e7fc640 10 monclient(hunting): dump: 2024-03-29T10:43:12.010 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:12.010 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:12.011 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:12.011 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:12.011 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:12.011 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:12.011 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:12.011 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:12.011 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:12.011 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:12.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.019+0000 7f511e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:12.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.019+0000 7f511e7fc640 1 -- 172.21.15.57:0/3849474291 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3532068635 0 0) 0x7f5128070e10 con 0x7f51280a9d70 2024-03-29T10:43:12.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.019+0000 7f511e7fc640 1 -- 172.21.15.57:0/3849474291 >> v1:172.21.15.57:6790/0 conn(0x7f512813f650 legacy=0x7f51281de9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:12.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.019+0000 7f511e7fc640 1 -- 172.21.15.57:0/3849474291 >> v1:172.21.15.179:6789/0 conn(0x7f51281457f0 legacy=0x7f51281e6d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:43:12.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.020+0000 7f511e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:43:12.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.020+0000 7f511e7fc640 1 monclient: found mon.a 2024-03-29T10:43:12.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.020+0000 7f511e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:12.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.020+0000 7f511e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:12.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.020+0000 7f511e7fc640 1 -- 172.21.15.57:0/3849474291 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f512806fbd0 con 0x7f51280a9d70 2024-03-29T10:43:12.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.020+0000 7f511e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:43:12.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.020+0000 7f511e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:43:12.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.020+0000 7f511e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:12.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.020+0000 7f511e7fc640 1 -- 172.21.15.57:0/3849474291 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51240044a0 con 0x7f51280a9d70 2024-03-29T10:43:12.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.020+0000 7f511e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:12.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.020+0000 7f511e7fc640 1 -- 172.21.15.57:0/3849474291 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f5124001070 con 0x7f51280a9d70 2024-03-29T10:43:12.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.020+0000 7f511e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:12.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.020+0000 7f511e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:12.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.020+0000 7f511e7fc640 10 monclient: dump: 2024-03-29T10:43:12.012 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:12.013 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:12.013 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:12.013 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:12.013 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:12.013 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:12.013 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:12.013 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:12.013 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:12.013 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:12.013 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.020+0000 7f511ffff640 4 set_mon_vals no callback set 2024-03-29T10:43:12.013 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.021+0000 7f512fb72640 5 monclient: authenticate success, global_id 4347 2024-03-29T10:43:12.013 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.021+0000 7f512fb72640 10 monclient: _renew_subs 2024-03-29T10:43:12.013 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.021+0000 7f512fb72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:12.013 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.021+0000 7f512fb72640 1 -- 172.21.15.57:0/3849474291 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5128070fc0 con 0x7f51280a9d70 2024-03-29T10:43:12.013 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.021+0000 7f512fb72640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:12.013 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.021+0000 7f512fb72640 10 monclient: _renew_subs 2024-03-29T10:43:12.014 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.022+0000 7f512fb72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:12.014 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.022+0000 7f512fb72640 1 -- 172.21.15.57:0/3849474291 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f51280705b0 con 0x7f51280a9d70 2024-03-29T10:43:12.014 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.022+0000 7f512fb72640 1 librados: init done 2024-03-29T10:43:12.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.024+0000 7f512fb72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:12.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.024+0000 7f512fb72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:12.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.024+0000 7f512fb72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:12.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.024+0000 7f512fb72640 1 -- 172.21.15.57:0/3849474291 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5128070a80 con 0x7f51280a9d70 2024-03-29T10:43:12.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.027+0000 7f511e7fc640 1 -- 172.21.15.57:0/3849474291 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f51240011d0 con 0x7f51280a9d70 2024-03-29T10:43:12.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.028+0000 7f512d10d640 10 monclient: get_auth_request con 0x7f510c06de80 auth_method 0 2024-03-29T10:43:12.016 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.028+0000 7f511e7fc640 1 -- 172.21.15.57:0/3849474291 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f5124084990 con 0x7f51280a9d70 2024-03-29T10:43:12.016 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.028+0000 7f511e7fc640 10 client.4347.objecter ms_dispatch 0x7f5128000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:12.016 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.028+0000 7f511e7fc640 3 client.4347.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:12.016 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.028+0000 7f511e7fc640 3 client.4347.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:12.016 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.028+0000 7f511e7fc640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-29T10:43:12.016 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.028+0000 7f511e7fc640 10 client.4347.objecter ms_handle_connect 0x7f510c06de80 2024-03-29T10:43:12.018 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.035+0000 7f511e7fc640 1 -- 172.21.15.57:0/3849474291 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5124050270 con 0x7f51280a9d70 2024-03-29T10:43:12.020 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.036+0000 7f511e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:12.020 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.036+0000 7f511e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:12.341 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.357+0000 7f512fb72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:12.341 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.357+0000 7f512fb72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:12.341 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.357+0000 7f512fb72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:12.341 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.357+0000 7f512fb72640 1 -- 172.21.15.57:0/3849474291 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5128071150 con 0x7f51280a9d70 2024-03-29T10:43:12.342 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.359+0000 7f511e7fc640 1 -- 172.21.15.57:0/3849474291 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f51240521a0 con 0x7f51280a9d70 2024-03-29T10:43:12.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.359+0000 7f511e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:12.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.359+0000 7f511e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:12.344 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:43:12.344 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:12.347 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.363+0000 7f512fb72640 10 librados: watch_flush enter 2024-03-29T10:43:12.347 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.363+0000 7f512fb72640 10 librados: watch_flush exit 2024-03-29T10:43:12.347 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.363+0000 7f512fb72640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-29T10:43:12.347 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.363+0000 7f512fb72640 10 client.4347.objecter successfully canceled tick 2024-03-29T10:43:12.348 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.364+0000 7f512fb72640 1 -- 172.21.15.57:0/3849474291 >> v1:172.21.15.179:6816/2698295936 conn(0x7f510c06de80 legacy=0x7f510c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:12.348 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.364+0000 7f512fb72640 10 monclient: shutdown 2024-03-29T10:43:12.348 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.364+0000 7f512fb72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:12.348 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.364+0000 7f512fb72640 1 -- 172.21.15.57:0/3849474291 >> v1:172.21.15.57:6789/0 conn(0x7f51280a9d70 legacy=0x7f51280b0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:12.348 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.364+0000 7f512fb72640 1 -- 172.21.15.57:0/3849474291 shutdown_connections 2024-03-29T10:43:12.348 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.364+0000 7f512fb72640 1 -- 172.21.15.57:0/3849474291 >> 172.21.15.57:0/3849474291 conn(0x7f5128066a00 msgr2=0x7f51280b4900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:12.348 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.364+0000 7f512fb72640 1 -- 172.21.15.57:0/3849474291 shutdown_connections 2024-03-29T10:43:12.348 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.364+0000 7f512fb72640 1 -- 172.21.15.57:0/3849474291 wait complete. 2024-03-29T10:43:12.348 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:12.364+0000 7f512fb72640 1 librados: shutdown 2024-03-29T10:43:12.365 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:19.172 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:19.316 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.332+0000 7f9566699640 10 monclient: get_monmap_and_config 2024-03-29T10:43:19.316 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.332+0000 7f9566699640 10 monclient: build_initial_monmap 2024-03-29T10:43:19.316 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.332+0000 7f9566699640 10 monclient: monmap: 2024-03-29T10:43:19.316 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:43:19.316 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:19.317 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:43:19.334210+0000 2024-03-29T10:43:19.317 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:43:19.334210+0000 2024-03-29T10:43:19.317 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:19.317 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:19.317 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:43:19.317 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:43:19.317 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:43:19.317 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:19.318 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.333+0000 7f9566699640 1 Processor -- start 2024-03-29T10:43:19.318 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.333+0000 7f9566699640 1 -- start start 2024-03-29T10:43:19.318 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.333+0000 7f9566699640 10 monclient: init 2024-03-29T10:43:19.318 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.334+0000 7f9566699640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:19.318 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.334+0000 7f9566699640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:43:19.318 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.334+0000 7f9566699640 10 monclient(hunting): picked mon.noname-c con 0x7f95600a9750 addr v1:172.21.15.179:6789/0 2024-03-29T10:43:19.318 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.334+0000 7f9566699640 10 monclient(hunting): picked mon.noname-a con 0x7f95600add40 addr v1:172.21.15.57:6789/0 2024-03-29T10:43:19.318 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.334+0000 7f9566699640 10 monclient(hunting): picked mon.noname-b con 0x7f95600aad20 addr v1:172.21.15.57:6790/0 2024-03-29T10:43:19.318 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.334+0000 7f9566699640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9560149870 con 0x7f95600add40 2024-03-29T10:43:19.318 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.334+0000 7f9566699640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95600abf80 con 0x7f95600a9750 2024-03-29T10:43:19.318 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.334+0000 7f9566699640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9560149ac0 con 0x7f95600aad20 2024-03-29T10:43:19.319 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.334+0000 7f9566699640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:19.319 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.334+0000 7f9566699640 10 monclient(hunting): authenticate will time out at 1151.511397s 2024-03-29T10:43:19.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.337+0000 7f9565697640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f95600a9750 0x7f9560149fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:38414/0 (socket says 172.21.15.57:38414) 2024-03-29T10:43:19.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.337+0000 7f9565697640 1 -- 172.21.15.57:0/1574961248 learned_addr learned my addr 172.21.15.57:0/1574961248 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:43:19.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.337+0000 7f9557fff640 1 -- 172.21.15.57:0/1574961248 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877265570 0 0) 0x7f9560149ac0 con 0x7f95600aad20 2024-03-29T10:43:19.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.337+0000 7f9557fff640 10 monclient(hunting): _negotiate 2024-03-29T10:43:19.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.337+0000 7f9557fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:19.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.337+0000 7f9557fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:19.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.337+0000 7f9557fff640 10 monclient(hunting): my global_id is 4373 2024-03-29T10:43:19.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.337+0000 7f9557fff640 1 -- 172.21.15.57:0/1574961248 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f954c003520 con 0x7f95600aad20 2024-03-29T10:43:19.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.339+0000 7f9557fff640 1 -- 172.21.15.57:0/1574961248 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1560804016 0 0) 0x7f954c003520 con 0x7f95600aad20 2024-03-29T10:43:19.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.339+0000 7f9557fff640 1 -- 172.21.15.57:0/1574961248 >> v1:172.21.15.179:6789/0 conn(0x7f95600a9750 legacy=0x7f9560149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:19.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.339+0000 7f9557fff640 1 -- 172.21.15.57:0/1574961248 >> v1:172.21.15.57:6789/0 conn(0x7f95600add40 legacy=0x7f9560143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:19.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.339+0000 7f9557fff640 10 monclient: _finish_hunting 0 2024-03-29T10:43:19.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.339+0000 7f9557fff640 1 monclient: found mon.noname-b 2024-03-29T10:43:19.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.339+0000 7f9557fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:19.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.339+0000 7f9557fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:43:19.324 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.339+0000 7f9557fff640 1 -- 172.21.15.57:0/1574961248 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956014a6f0 con 0x7f95600aad20 2024-03-29T10:43:19.324 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.339+0000 7f9557fff640 10 monclient: _finish_auth 0 2024-03-29T10:43:19.324 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.339+0000 7f9557fff640 10 monclient: _check_auth_tickets 2024-03-29T10:43:19.324 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.339+0000 7f9557fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:19.324 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.339+0000 7f9557fff640 1 -- 172.21.15.57:0/1574961248 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f95580019f0 con 0x7f95600aad20 2024-03-29T10:43:19.324 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.339+0000 7f9557fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:19.324 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.339+0000 7f9557fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:19.324 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.339+0000 7f9557fff640 10 monclient: dump: 2024-03-29T10:43:19.324 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:19.324 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:19.324 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:19.324 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:19.324 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:19.324 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:19.324 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:19.324 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:19.325 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:19.325 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:19.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.339+0000 7f9566699640 5 monclient: authenticate success, global_id 4373 2024-03-29T10:43:19.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.339+0000 7f9566699640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:19.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.340+0000 7f9557fff640 1 -- 172.21.15.57:0/1574961248 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9558001bc0 con 0x7f95600aad20 2024-03-29T10:43:19.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.340+0000 7f9557fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:19.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.340+0000 7f9566699640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:19.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.340+0000 7f9566699640 4 set_mon_vals no callback set 2024-03-29T10:43:19.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.340+0000 7f9557fff640 1 -- 172.21.15.57:0/1574961248 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f9558001fd0 con 0x7f95600aad20 2024-03-29T10:43:19.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.340+0000 7f9557fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:19.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.340+0000 7f9557fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:19.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.340+0000 7f9557fff640 10 monclient: dump: 2024-03-29T10:43:19.326 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:19.326 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:19.326 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:19.326 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:19.326 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:19.327 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:19.327 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:19.327 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:19.327 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:19.327 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:19.327 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.341+0000 7f9566699640 10 monclient: shutdown 2024-03-29T10:43:19.327 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.341+0000 7f9566699640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:19.327 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.341+0000 7f9566699640 1 -- 172.21.15.57:0/1574961248 >> v1:172.21.15.57:6790/0 conn(0x7f95600aad20 legacy=0x7f9560151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:19.327 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.341+0000 7f9566699640 1 -- 172.21.15.57:0/1574961248 shutdown_connections 2024-03-29T10:43:19.327 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.341+0000 7f9566699640 1 -- 172.21.15.57:0/1574961248 >> 172.21.15.57:0/1574961248 conn(0x7f95600673e0 msgr2=0x7f95600b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:19.327 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.342+0000 7f9566699640 1 -- 172.21.15.57:0/1574961248 shutdown_connections 2024-03-29T10:43:19.327 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.342+0000 7f9566699640 1 -- 172.21.15.57:0/1574961248 wait complete. 2024-03-29T10:43:19.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.342+0000 7f9566699640 10 monclient: build_initial_monmap 2024-03-29T10:43:19.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.343+0000 7f9566699640 10 monclient: monmap: 2024-03-29T10:43:19.329 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:43:19.329 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:19.329 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:43:19.329 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:43:19.329 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:19.329 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:19.329 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:43:19.329 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:43:19.330 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:43:19.330 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:19.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.343+0000 7f9566699640 1 librados: starting msgr at 2024-03-29T10:43:19.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.343+0000 7f9566699640 1 librados: starting objecter 2024-03-29T10:43:19.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.343+0000 7f9566699640 1 Processor -- start 2024-03-29T10:43:19.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.343+0000 7f9566699640 1 -- start start 2024-03-29T10:43:19.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.343+0000 7f9566699640 1 librados: setting wanted keys 2024-03-29T10:43:19.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.343+0000 7f9566699640 1 librados: calling monclient init 2024-03-29T10:43:19.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.343+0000 7f9566699640 10 monclient: init 2024-03-29T10:43:19.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.344+0000 7f9566699640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:19.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.344+0000 7f9566699640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:43:19.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.344+0000 7f9566699640 10 monclient(hunting): picked mon.noname-c con 0x7f95600a9750 addr v1:172.21.15.57:6790/0 2024-03-29T10:43:19.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.344+0000 7f9566699640 10 monclient(hunting): picked mon.noname-a con 0x7f95600aad20 addr v1:172.21.15.57:6789/0 2024-03-29T10:43:19.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.344+0000 7f9566699640 10 monclient(hunting): picked mon.noname-b con 0x7f95600add40 addr v1:172.21.15.179:6789/0 2024-03-29T10:43:19.331 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.344+0000 7f9566699640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9560134a40 con 0x7f95600aad20 2024-03-29T10:43:19.331 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.344+0000 7f9566699640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9560149750 con 0x7f95600add40 2024-03-29T10:43:19.331 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.344+0000 7f9566699640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9560146fc0 con 0x7f95600a9750 2024-03-29T10:43:19.331 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.344+0000 7f9566699640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:19.331 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.344+0000 7f9566699640 10 monclient(hunting): authenticate will time out at 1151.521494s 2024-03-29T10:43:19.331 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.345+0000 7f9565e98640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f95600add40 0x7f95601ffbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:38832/0 (socket says 172.21.15.57:38832) 2024-03-29T10:43:19.331 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.345+0000 7f9565e98640 1 -- 172.21.15.57:0/2000550463 learned_addr learned my addr 172.21.15.57:0/2000550463 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:43:19.331 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.345+0000 7f9555ffb640 10 client.?.objecter ms_handle_connect 0x7f95600add40 2024-03-29T10:43:19.331 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.345+0000 7f9555ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:19.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.346+0000 7f9555ffb640 1 -- 172.21.15.57:0/2000550463 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3030305595 0 0) 0x7f9560149750 con 0x7f95600add40 2024-03-29T10:43:19.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.346+0000 7f9555ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:43:19.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.346+0000 7f9555ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:19.333 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.346+0000 7f9555ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:19.333 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.346+0000 7f9555ffb640 10 monclient(hunting): my global_id is 4357 2024-03-29T10:43:19.333 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.346+0000 7f9555ffb640 1 -- 172.21.15.57:0/2000550463 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9544004eb0 con 0x7f95600add40 2024-03-29T10:43:19.333 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.346+0000 7f9555ffb640 1 -- 172.21.15.57:0/2000550463 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3782925236 0 0) 0x7f9544004eb0 con 0x7f95600add40 2024-03-29T10:43:19.333 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.346+0000 7f9555ffb640 1 -- 172.21.15.57:0/2000550463 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9560149750 con 0x7f95600add40 2024-03-29T10:43:19.333 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.346+0000 7f9555ffb640 1 -- 172.21.15.57:0/2000550463 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f9558001fd0 con 0x7f95600add40 2024-03-29T10:43:19.333 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.346+0000 7f9555ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:19.333 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.346+0000 7f9555ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:19.333 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.346+0000 7f9555ffb640 10 monclient(hunting): dump: 2024-03-29T10:43:19.333 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:19.333 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:19.333 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:19.333 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:19.333 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:19.334 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:19.334 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:19.334 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:19.334 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:19.334 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:19.334 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.346+0000 7f9555ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:19.334 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.346+0000 7f9555ffb640 1 -- 172.21.15.57:0/2000550463 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2968332926 0 0) 0x7f9560149750 con 0x7f95600add40 2024-03-29T10:43:19.334 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.346+0000 7f9555ffb640 1 -- 172.21.15.57:0/2000550463 >> v1:172.21.15.57:6790/0 conn(0x7f95600a9750 legacy=0x7f95601485a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:43:19.334 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.350+0000 7f9555ffb640 1 -- 172.21.15.57:0/2000550463 >> v1:172.21.15.57:6789/0 conn(0x7f95600aad20 legacy=0x7f9560135370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:43:19.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.352+0000 7f9555ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:43:19.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.352+0000 7f9555ffb640 1 monclient: found mon.b 2024-03-29T10:43:19.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.352+0000 7f9555ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:19.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.352+0000 7f9555ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:43:19.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.352+0000 7f9555ffb640 1 -- 172.21.15.57:0/2000550463 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9560147150 con 0x7f95600add40 2024-03-29T10:43:19.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.352+0000 7f9555ffb640 10 monclient: _finish_auth 0 2024-03-29T10:43:19.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.352+0000 7f9555ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:43:19.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.352+0000 7f9555ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:19.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.353+0000 7f9555ffb640 1 -- 172.21.15.57:0/2000550463 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9558003c20 con 0x7f95600add40 2024-03-29T10:43:19.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.353+0000 7f9555ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:19.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.353+0000 7f9556ffd640 4 set_mon_vals no callback set 2024-03-29T10:43:19.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.354+0000 7f9555ffb640 1 -- 172.21.15.57:0/2000550463 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f9558004070 con 0x7f95600add40 2024-03-29T10:43:19.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.354+0000 7f9555ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:19.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.354+0000 7f9555ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:19.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.354+0000 7f9555ffb640 10 monclient: dump: 2024-03-29T10:43:19.340 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:19.340 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:19.340 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:19.340 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:19.340 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:19.340 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:19.340 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:19.340 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:19.340 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:19.340 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:19.340 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.354+0000 7f9566699640 5 monclient: authenticate success, global_id 4357 2024-03-29T10:43:19.340 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.354+0000 7f9566699640 10 monclient: _renew_subs 2024-03-29T10:43:19.340 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.354+0000 7f9566699640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:43:19.340 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.354+0000 7f9566699640 1 -- 172.21.15.57:0/2000550463 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9560148cb0 con 0x7f95600add40 2024-03-29T10:43:19.341 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.354+0000 7f9566699640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:19.341 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.354+0000 7f9566699640 10 monclient: _renew_subs 2024-03-29T10:43:19.341 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.354+0000 7f9566699640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:43:19.341 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.354+0000 7f9566699640 1 -- 172.21.15.57:0/2000550463 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9560134600 con 0x7f95600add40 2024-03-29T10:43:19.341 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.354+0000 7f9566699640 1 librados: init done 2024-03-29T10:43:19.341 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.356+0000 7f9566699640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:19.342 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.356+0000 7f9566699640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:19.342 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.356+0000 7f9566699640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:43:19.342 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.356+0000 7f9566699640 1 -- 172.21.15.57:0/2000550463 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f956004f730 con 0x7f95600add40 2024-03-29T10:43:19.342 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.357+0000 7f9555ffb640 1 -- 172.21.15.57:0/2000550463 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f95580041d0 con 0x7f95600add40 2024-03-29T10:43:19.342 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.358+0000 7f9555ffb640 1 -- 172.21.15.57:0/2000550463 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f9558084f80 con 0x7f95600add40 2024-03-29T10:43:19.342 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.358+0000 7f9555ffb640 10 client.4357.objecter ms_dispatch 0x7f9560000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:19.342 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.358+0000 7f9555ffb640 3 client.4357.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:19.342 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.358+0000 7f9555ffb640 3 client.4357.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:19.342 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.358+0000 7f9555ffb640 20 client.4357.objecter dump_active .. 0 homeless 2024-03-29T10:43:19.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.358+0000 7f9565697640 10 monclient: get_auth_request con 0x7f954406dd20 auth_method 0 2024-03-29T10:43:19.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.359+0000 7f9555ffb640 10 client.4357.objecter ms_handle_connect 0x7f954406dd20 2024-03-29T10:43:19.350 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.366+0000 7f9555ffb640 1 -- 172.21.15.57:0/2000550463 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9558050860 con 0x7f95600add40 2024-03-29T10:43:19.350 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.366+0000 7f9555ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:19.350 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.366+0000 7f9555ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:19.694 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.710+0000 7f9566699640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:19.694 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.710+0000 7f9566699640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:19.694 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.710+0000 7f9566699640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:43:19.694 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.710+0000 7f9566699640 1 -- 172.21.15.57:0/2000550463 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9560134bd0 con 0x7f95600add40 2024-03-29T10:43:19.695 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.711+0000 7f9555ffb640 1 -- 172.21.15.57:0/2000550463 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f9558052790 con 0x7f95600add40 2024-03-29T10:43:19.695 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.711+0000 7f9555ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:19.695 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.711+0000 7f9555ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:19.695 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:43:19.695 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:19.698 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.714+0000 7f9566699640 10 librados: watch_flush enter 2024-03-29T10:43:19.698 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.714+0000 7f9566699640 10 librados: watch_flush exit 2024-03-29T10:43:19.698 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.714+0000 7f9566699640 20 client.4357.objecter shutdown clearing up homeless session... 2024-03-29T10:43:19.698 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.714+0000 7f9566699640 10 client.4357.objecter successfully canceled tick 2024-03-29T10:43:19.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.715+0000 7f9566699640 1 -- 172.21.15.57:0/2000550463 >> v1:172.21.15.179:6816/2698295936 conn(0x7f954406dd20 legacy=0x7f95440701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:19.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.715+0000 7f9566699640 10 monclient: shutdown 2024-03-29T10:43:19.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.715+0000 7f9566699640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:19.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.715+0000 7f9566699640 1 -- 172.21.15.57:0/2000550463 >> v1:172.21.15.179:6789/0 conn(0x7f95600add40 legacy=0x7f95601ffbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:19.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.715+0000 7f9566699640 1 -- 172.21.15.57:0/2000550463 shutdown_connections 2024-03-29T10:43:19.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.715+0000 7f9566699640 1 -- 172.21.15.57:0/2000550463 >> 172.21.15.57:0/2000550463 conn(0x7f95600673e0 msgr2=0x7f9560069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:19.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.715+0000 7f9566699640 1 -- 172.21.15.57:0/2000550463 shutdown_connections 2024-03-29T10:43:19.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.715+0000 7f9566699640 1 -- 172.21.15.57:0/2000550463 wait complete. 2024-03-29T10:43:19.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:19.715+0000 7f9566699640 1 librados: shutdown 2024-03-29T10:43:19.716 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:26.522 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:26.665 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.681+0000 7f1e6ffe2640 10 monclient: get_monmap_and_config 2024-03-29T10:43:26.665 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.681+0000 7f1e6ffe2640 10 monclient: build_initial_monmap 2024-03-29T10:43:26.665 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.681+0000 7f1e6ffe2640 10 monclient: monmap: 2024-03-29T10:43:26.665 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:43:26.665 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:26.666 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:43:26.682849+0000 2024-03-29T10:43:26.666 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:43:26.682849+0000 2024-03-29T10:43:26.666 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:26.666 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:26.666 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:43:26.666 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:43:26.666 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:43:26.666 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:26.666 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.682+0000 7f1e6ffe2640 1 Processor -- start 2024-03-29T10:43:26.667 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.683+0000 7f1e6ffe2640 1 -- start start 2024-03-29T10:43:26.667 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.683+0000 7f1e6ffe2640 10 monclient: init 2024-03-29T10:43:26.669 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.684+0000 7f1e6ffe2640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:26.669 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.684+0000 7f1e6ffe2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:43:26.669 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.684+0000 7f1e6ffe2640 10 monclient(hunting): picked mon.noname-b con 0x7f1e680b4330 addr v1:172.21.15.57:6790/0 2024-03-29T10:43:26.669 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.684+0000 7f1e6ffe2640 10 monclient(hunting): picked mon.noname-a con 0x7f1e681470b0 addr v1:172.21.15.57:6789/0 2024-03-29T10:43:26.669 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.684+0000 7f1e6ffe2640 10 monclient(hunting): picked mon.noname-c con 0x7f1e681439b0 addr v1:172.21.15.179:6789/0 2024-03-29T10:43:26.669 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.684+0000 7f1e6ffe2640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e6806f2a0 con 0x7f1e681470b0 2024-03-29T10:43:26.669 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.684+0000 7f1e6ffe2640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e68146b90 con 0x7f1e681439b0 2024-03-29T10:43:26.669 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.684+0000 7f1e6ffe2640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e680b2ca0 con 0x7f1e680b4330 2024-03-29T10:43:26.669 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.684+0000 7f1e6ffe2640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:26.669 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.684+0000 7f1e6ffe2640 10 monclient(hunting): authenticate will time out at 1158.861443s 2024-03-29T10:43:26.669 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.684+0000 7f1e6e57f640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f1e681439b0 0x7f1e68151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:48516/0 (socket says 172.21.15.57:48516) 2024-03-29T10:43:26.669 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.684+0000 7f1e6e57f640 1 -- 172.21.15.57:0/295700064 learned_addr learned my addr 172.21.15.57:0/295700064 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:43:26.669 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.684+0000 7f1e6cd7c640 1 -- 172.21.15.57:0/295700064 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 306334628 0 0) 0x7f1e68146b90 con 0x7f1e681439b0 2024-03-29T10:43:26.669 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.685+0000 7f1e6cd7c640 10 monclient(hunting): _negotiate 2024-03-29T10:43:26.670 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.685+0000 7f1e6cd7c640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:26.670 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.685+0000 7f1e6cd7c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:26.670 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.685+0000 7f1e6cd7c640 10 monclient(hunting): my global_id is 4360 2024-03-29T10:43:26.670 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.685+0000 7f1e6cd7c640 1 -- 172.21.15.57:0/295700064 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e54003520 con 0x7f1e681439b0 2024-03-29T10:43:26.670 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.685+0000 7f1e6cd7c640 1 -- 172.21.15.57:0/295700064 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1834883012 0 0) 0x7f1e54003520 con 0x7f1e681439b0 2024-03-29T10:43:26.670 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.685+0000 7f1e6cd7c640 1 -- 172.21.15.57:0/295700064 >> v1:172.21.15.57:6790/0 conn(0x7f1e680b4330 legacy=0x7f1e680b4710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:43:26.672 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.688+0000 7f1e6cd7c640 1 -- 172.21.15.57:0/295700064 >> v1:172.21.15.57:6789/0 conn(0x7f1e681470b0 legacy=0x7f1e680b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:26.672 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.688+0000 7f1e6cd7c640 10 monclient: _finish_hunting 0 2024-03-29T10:43:26.672 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.688+0000 7f1e6cd7c640 1 monclient: found mon.noname-c 2024-03-29T10:43:26.672 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.688+0000 7f1e6cd7c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:26.672 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.688+0000 7f1e6cd7c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:43:26.672 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.688+0000 7f1e6cd7c640 1 -- 172.21.15.57:0/295700064 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6806e0c0 con 0x7f1e681439b0 2024-03-29T10:43:26.672 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.688+0000 7f1e6cd7c640 10 monclient: _finish_auth 0 2024-03-29T10:43:26.672 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.688+0000 7f1e6cd7c640 10 monclient: _check_auth_tickets 2024-03-29T10:43:26.673 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.688+0000 7f1e6cd7c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:26.673 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.688+0000 7f1e6ffe2640 5 monclient: authenticate success, global_id 4360 2024-03-29T10:43:26.673 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.688+0000 7f1e6ffe2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:26.673 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.688+0000 7f1e6cd7c640 1 -- 172.21.15.57:0/295700064 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1e60001a10 con 0x7f1e681439b0 2024-03-29T10:43:26.673 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.688+0000 7f1e6cd7c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:26.673 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.688+0000 7f1e6cd7c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:26.673 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.688+0000 7f1e6cd7c640 10 monclient: dump: 2024-03-29T10:43:26.673 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:26.673 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:26.673 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:26.673 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:26.673 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:26.673 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:26.673 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:26.673 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:26.673 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:26.673 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:26.674 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.688+0000 7f1e6ffe2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:26.674 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.689+0000 7f1e6cd7c640 1 -- 172.21.15.57:0/295700064 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e60001ca0 con 0x7f1e681439b0 2024-03-29T10:43:26.674 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.689+0000 7f1e6cd7c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:26.674 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.689+0000 7f1e6ffe2640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:26.674 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.689+0000 7f1e6ffe2640 4 set_mon_vals no callback set 2024-03-29T10:43:26.674 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.690+0000 7f1e6ffe2640 10 monclient: shutdown 2024-03-29T10:43:26.674 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.690+0000 7f1e6ffe2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:26.675 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.690+0000 7f1e6ffe2640 1 -- 172.21.15.57:0/295700064 >> v1:172.21.15.179:6789/0 conn(0x7f1e681439b0 legacy=0x7f1e68151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:26.675 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.691+0000 7f1e6ffe2640 1 -- 172.21.15.57:0/295700064 shutdown_connections 2024-03-29T10:43:26.675 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.691+0000 7f1e6ffe2640 1 -- 172.21.15.57:0/295700064 >> 172.21.15.57:0/295700064 conn(0x7f1e68066a00 msgr2=0x7f1e68066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:26.677 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.693+0000 7f1e6ffe2640 1 -- 172.21.15.57:0/295700064 shutdown_connections 2024-03-29T10:43:26.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.693+0000 7f1e6ffe2640 1 -- 172.21.15.57:0/295700064 wait complete. 2024-03-29T10:43:26.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.694+0000 7f1e6ffe2640 10 monclient: build_initial_monmap 2024-03-29T10:43:26.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.694+0000 7f1e6ffe2640 10 monclient: monmap: 2024-03-29T10:43:26.678 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:43:26.679 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:26.679 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:43:26.679 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:43:26.679 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:26.679 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:26.679 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:43:26.679 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:43:26.679 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:43:26.679 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:26.679 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.695+0000 7f1e6ffe2640 1 librados: starting msgr at 2024-03-29T10:43:26.679 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.695+0000 7f1e6ffe2640 1 librados: starting objecter 2024-03-29T10:43:26.679 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.695+0000 7f1e6ffe2640 1 Processor -- start 2024-03-29T10:43:26.680 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.695+0000 7f1e6ffe2640 1 -- start start 2024-03-29T10:43:26.680 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.695+0000 7f1e6ffe2640 1 librados: setting wanted keys 2024-03-29T10:43:26.680 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.695+0000 7f1e6ffe2640 1 librados: calling monclient init 2024-03-29T10:43:26.680 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.695+0000 7f1e6ffe2640 10 monclient: init 2024-03-29T10:43:26.680 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.696+0000 7f1e6ffe2640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:26.680 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.696+0000 7f1e6ffe2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:43:26.680 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.696+0000 7f1e6ffe2640 10 monclient(hunting): picked mon.noname-b con 0x7f1e680b4330 addr v1:172.21.15.179:6789/0 2024-03-29T10:43:26.683 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.699+0000 7f1e6ffe2640 10 monclient(hunting): picked mon.noname-c con 0x7f1e681439b0 addr v1:172.21.15.57:6790/0 2024-03-29T10:43:26.683 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.699+0000 7f1e6ffe2640 10 monclient(hunting): picked mon.noname-a con 0x7f1e681470b0 addr v1:172.21.15.57:6789/0 2024-03-29T10:43:26.683 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.699+0000 7f1e6ffe2640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e681dfbd0 con 0x7f1e681470b0 2024-03-29T10:43:26.683 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.699+0000 7f1e6ffe2640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e68121bc0 con 0x7f1e680b4330 2024-03-29T10:43:26.683 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.699+0000 7f1e6ffe2640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e680b7860 con 0x7f1e681439b0 2024-03-29T10:43:26.683 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.699+0000 7f1e6ffe2640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:26.683 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.699+0000 7f1e6ffe2640 10 monclient(hunting): authenticate will time out at 1158.876328s 2024-03-29T10:43:26.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.699+0000 7f1e6dd7e640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f1e680b4330 0x7f1e681def10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:48982/0 (socket says 172.21.15.57:48982) 2024-03-29T10:43:26.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.699+0000 7f1e6dd7e640 1 -- 172.21.15.57:0/663449532 learned_addr learned my addr 172.21.15.57:0/663449532 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:43:26.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.700+0000 7f1e5e7fc640 10 client.?.objecter ms_handle_connect 0x7f1e681439b0 2024-03-29T10:43:26.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.700+0000 7f1e5e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:26.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.700+0000 7f1e5e7fc640 1 -- 172.21.15.57:0/663449532 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479729474 0 0) 0x7f1e680b7860 con 0x7f1e681439b0 2024-03-29T10:43:26.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.700+0000 7f1e5e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:26.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.700+0000 7f1e5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:26.685 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.700+0000 7f1e5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:26.685 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.700+0000 7f1e5e7fc640 10 monclient(hunting): my global_id is 4376 2024-03-29T10:43:26.685 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.700+0000 7f1e5e7fc640 1 -- 172.21.15.57:0/663449532 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e4c004eb0 con 0x7f1e681439b0 2024-03-29T10:43:26.685 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.701+0000 7f1e5e7fc640 1 -- 172.21.15.57:0/663449532 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3103681587 0 0) 0x7f1e4c004eb0 con 0x7f1e681439b0 2024-03-29T10:43:26.685 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.701+0000 7f1e5e7fc640 1 -- 172.21.15.57:0/663449532 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e680b7860 con 0x7f1e681439b0 2024-03-29T10:43:26.685 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.701+0000 7f1e5e7fc640 10 client.?.objecter ms_handle_connect 0x7f1e680b4330 2024-03-29T10:43:26.685 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.701+0000 7f1e5e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:26.685 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.701+0000 7f1e5e7fc640 1 -- 172.21.15.57:0/663449532 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1e580018c0 con 0x7f1e681439b0 2024-03-29T10:43:26.686 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.701+0000 7f1e5e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:26.686 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.701+0000 7f1e5e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:26.686 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.701+0000 7f1e5e7fc640 10 monclient(hunting): dump: 2024-03-29T10:43:26.686 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:26.686 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:26.686 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:26.686 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:26.686 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:26.686 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:26.686 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:26.686 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:26.686 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:26.686 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:26.686 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.701+0000 7f1e5e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:26.686 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.701+0000 7f1e5e7fc640 1 -- 172.21.15.57:0/663449532 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2112941501 0 0) 0x7f1e680b7860 con 0x7f1e681439b0 2024-03-29T10:43:26.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.701+0000 7f1e5e7fc640 1 -- 172.21.15.57:0/663449532 >> v1:172.21.15.179:6789/0 conn(0x7f1e680b4330 legacy=0x7f1e681def10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:26.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.701+0000 7f1e5e7fc640 1 -- 172.21.15.57:0/663449532 >> v1:172.21.15.57:6789/0 conn(0x7f1e681470b0 legacy=0x7f1e6811fae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:43:26.689 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.703+0000 7f1e5e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:43:26.689 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.703+0000 7f1e5e7fc640 1 monclient: found mon.c 2024-03-29T10:43:26.689 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.703+0000 7f1e5e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:26.689 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.703+0000 7f1e5e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:43:26.689 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.703+0000 7f1e5e7fc640 1 -- 172.21.15.57:0/663449532 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e681df620 con 0x7f1e681439b0 2024-03-29T10:43:26.689 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.703+0000 7f1e5e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:43:26.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.703+0000 7f1e5e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:43:26.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.703+0000 7f1e5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:26.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.703+0000 7f1e6ffe2640 5 monclient: authenticate success, global_id 4376 2024-03-29T10:43:26.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.703+0000 7f1e6ffe2640 10 monclient: _renew_subs 2024-03-29T10:43:26.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.703+0000 7f1e6ffe2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:43:26.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.703+0000 7f1e6ffe2640 1 -- 172.21.15.57:0/663449532 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e681df8d0 con 0x7f1e681439b0 2024-03-29T10:43:26.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.703+0000 7f1e6ffe2640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:26.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.703+0000 7f1e6ffe2640 10 monclient: _renew_subs 2024-03-29T10:43:26.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.703+0000 7f1e6ffe2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:43:26.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.703+0000 7f1e6ffe2640 1 -- 172.21.15.57:0/663449532 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e68140f90 con 0x7f1e681439b0 2024-03-29T10:43:26.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.703+0000 7f1e6ffe2640 1 librados: init done 2024-03-29T10:43:26.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.703+0000 7f1e5e7fc640 1 -- 172.21.15.57:0/663449532 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e580035f0 con 0x7f1e681439b0 2024-03-29T10:43:26.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.703+0000 7f1e5e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:26.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.703+0000 7f1e5ffff640 4 set_mon_vals no callback set 2024-03-29T10:43:26.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.704+0000 7f1e5e7fc640 1 -- 172.21.15.57:0/663449532 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1e58003a40 con 0x7f1e681439b0 2024-03-29T10:43:26.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.704+0000 7f1e5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:26.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.704+0000 7f1e5e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:26.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.704+0000 7f1e5e7fc640 10 monclient: dump: 2024-03-29T10:43:26.691 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:26.691 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:26.691 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:26.691 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:26.691 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:26.691 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:26.691 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:26.691 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:26.691 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:26.691 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:26.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.705+0000 7f1e4bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:26.692 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.705+0000 7f1e4bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:26.692 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.705+0000 7f1e4bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:43:26.692 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.705+0000 7f1e4bfff640 1 -- 172.21.15.57:0/663449532 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e44005080 con 0x7f1e681439b0 2024-03-29T10:43:26.693 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.706+0000 7f1e5e7fc640 1 -- 172.21.15.57:0/663449532 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f1e58003c20 con 0x7f1e681439b0 2024-03-29T10:43:26.693 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.707+0000 7f1e5e7fc640 1 -- 172.21.15.57:0/663449532 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f1e58083960 con 0x7f1e681439b0 2024-03-29T10:43:26.693 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.707+0000 7f1e5e7fc640 10 client.4376.objecter ms_dispatch 0x7f1e68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:26.693 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.707+0000 7f1e5e7fc640 3 client.4376.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:26.693 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.707+0000 7f1e5e7fc640 3 client.4376.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:26.693 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.707+0000 7f1e5e7fc640 20 client.4376.objecter dump_active .. 0 homeless 2024-03-29T10:43:26.693 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.707+0000 7f1e6dd7e640 10 monclient: get_auth_request con 0x7f1e4c06dbf0 auth_method 0 2024-03-29T10:43:26.693 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.708+0000 7f1e5e7fc640 10 client.4376.objecter ms_handle_connect 0x7f1e4c06dbf0 2024-03-29T10:43:26.698 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.715+0000 7f1e5e7fc640 1 -- 172.21.15.57:0/663449532 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1e5804f240 con 0x7f1e681439b0 2024-03-29T10:43:26.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.715+0000 7f1e5e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:26.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:26.715+0000 7f1e5e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:27.025 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:27.041+0000 7f1e4bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:27.025 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:27.041+0000 7f1e4bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:27.025 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:27.041+0000 7f1e4bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:43:27.025 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:27.041+0000 7f1e4bfff640 1 -- 172.21.15.57:0/663449532 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e44005330 con 0x7f1e681439b0 2024-03-29T10:43:27.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:27.042+0000 7f1e5e7fc640 1 -- 172.21.15.57:0/663449532 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f1e58051170 con 0x7f1e681439b0 2024-03-29T10:43:27.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:27.042+0000 7f1e5e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:27.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:27.042+0000 7f1e5e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:27.026 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:43:27.026 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:27.029 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:27.045+0000 7f1e4bfff640 10 librados: watch_flush enter 2024-03-29T10:43:27.029 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:27.045+0000 7f1e4bfff640 10 librados: watch_flush exit 2024-03-29T10:43:27.029 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:27.045+0000 7f1e4bfff640 20 client.4376.objecter shutdown clearing up homeless session... 2024-03-29T10:43:27.030 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:27.045+0000 7f1e4bfff640 10 client.4376.objecter successfully canceled tick 2024-03-29T10:43:27.030 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:27.046+0000 7f1e4bfff640 1 -- 172.21.15.57:0/663449532 >> v1:172.21.15.179:6816/2698295936 conn(0x7f1e4c06dbf0 legacy=0x7f1e4c0700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:27.030 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:27.046+0000 7f1e4bfff640 10 monclient: shutdown 2024-03-29T10:43:27.030 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:27.046+0000 7f1e4bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:27.030 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:27.046+0000 7f1e4bfff640 1 -- 172.21.15.57:0/663449532 >> v1:172.21.15.57:6790/0 conn(0x7f1e681439b0 legacy=0x7f1e681406a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:27.030 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:27.046+0000 7f1e4bfff640 1 -- 172.21.15.57:0/663449532 shutdown_connections 2024-03-29T10:43:27.030 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:27.046+0000 7f1e4bfff640 1 -- 172.21.15.57:0/663449532 >> 172.21.15.57:0/663449532 conn(0x7f1e68066a00 msgr2=0x7f1e68066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:27.031 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:27.046+0000 7f1e4bfff640 1 -- 172.21.15.57:0/663449532 shutdown_connections 2024-03-29T10:43:27.031 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:27.046+0000 7f1e4bfff640 1 -- 172.21.15.57:0/663449532 wait complete. 2024-03-29T10:43:27.031 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:27.047+0000 7f1e4bfff640 1 librados: shutdown 2024-03-29T10:43:27.048 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:33.854 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:34.009 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.025+0000 7f301158e640 10 monclient: get_monmap_and_config 2024-03-29T10:43:34.009 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.025+0000 7f301158e640 10 monclient: build_initial_monmap 2024-03-29T10:43:34.009 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.025+0000 7f301158e640 10 monclient: monmap: 2024-03-29T10:43:34.009 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:43:34.009 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:34.009 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:43:34.026534+0000 2024-03-29T10:43:34.009 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:43:34.026534+0000 2024-03-29T10:43:34.009 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:34.010 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:34.010 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:43:34.010 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:43:34.010 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:43:34.010 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:34.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.027+0000 7f301158e640 1 Processor -- start 2024-03-29T10:43:34.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.027+0000 7f301158e640 1 -- start start 2024-03-29T10:43:34.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.027+0000 7f301158e640 10 monclient: init 2024-03-29T10:43:34.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.028+0000 7f301158e640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:34.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.028+0000 7f301158e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:43:34.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.028+0000 7f301158e640 10 monclient(hunting): picked mon.noname-a con 0x7f300c0b0860 addr v1:172.21.15.57:6789/0 2024-03-29T10:43:34.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.028+0000 7f301158e640 10 monclient(hunting): picked mon.noname-c con 0x7f300c13ef30 addr v1:172.21.15.179:6789/0 2024-03-29T10:43:34.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.028+0000 7f301158e640 10 monclient(hunting): picked mon.noname-b con 0x7f300c0af8e0 addr v1:172.21.15.57:6790/0 2024-03-29T10:43:34.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.028+0000 7f301158e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f300c0b0c40 con 0x7f300c0b0860 2024-03-29T10:43:34.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.028+0000 7f301158e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f300c0aed10 con 0x7f300c13ef30 2024-03-29T10:43:34.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.028+0000 7f301158e640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f300c0afd50 con 0x7f300c0af8e0 2024-03-29T10:43:34.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.028+0000 7f301158e640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:34.012 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.028+0000 7f301158e640 10 monclient(hunting): authenticate will time out at 1166.204631s 2024-03-29T10:43:34.014 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.031+0000 7f300b7fe640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f300c0af8e0 0x7f300c149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:38501/0 (socket says 172.21.15.57:38501) 2024-03-29T10:43:34.014 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.031+0000 7f300b7fe640 1 -- 172.21.15.57:0/3834999762 learned_addr learned my addr 172.21.15.57:0/3834999762 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:43:34.018 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.034+0000 7f3009ffb640 1 -- 172.21.15.57:0/3834999762 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285413468 0 0) 0x7f300c0afd50 con 0x7f300c0af8e0 2024-03-29T10:43:34.018 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.034+0000 7f3009ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:43:34.018 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.034+0000 7f3009ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:34.018 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.034+0000 7f3009ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:34.018 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.034+0000 7f3009ffb640 10 monclient(hunting): my global_id is 4379 2024-03-29T10:43:34.018 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.034+0000 7f3009ffb640 1 -- 172.21.15.57:0/3834999762 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ff4003520 con 0x7f300c0af8e0 2024-03-29T10:43:34.019 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.034+0000 7f3009ffb640 1 -- 172.21.15.57:0/3834999762 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102783482 0 0) 0x7f300c0aed10 con 0x7f300c13ef30 2024-03-29T10:43:34.019 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.034+0000 7f3009ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:43:34.019 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.034+0000 7f3009ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:34.019 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.034+0000 7f3009ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:34.019 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.034+0000 7f3009ffb640 10 monclient(hunting): my global_id is 4363 2024-03-29T10:43:34.019 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.034+0000 7f3009ffb640 1 -- 172.21.15.57:0/3834999762 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f300c0afd50 con 0x7f300c13ef30 2024-03-29T10:43:34.019 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.035+0000 7f3009ffb640 1 -- 172.21.15.57:0/3834999762 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 709879899 0 0) 0x7f2ff4003520 con 0x7f300c0af8e0 2024-03-29T10:43:34.019 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.035+0000 7f3009ffb640 1 -- 172.21.15.57:0/3834999762 >> v1:172.21.15.179:6789/0 conn(0x7f300c13ef30 legacy=0x7f300c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:34.019 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.035+0000 7f3009ffb640 1 -- 172.21.15.57:0/3834999762 >> v1:172.21.15.57:6789/0 conn(0x7f300c0b0860 legacy=0x7f300c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:34.019 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.035+0000 7f3009ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:43:34.019 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.035+0000 7f3009ffb640 1 monclient: found mon.noname-b 2024-03-29T10:43:34.019 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.035+0000 7f3009ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:34.019 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.035+0000 7f3009ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:43:34.020 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.035+0000 7f3009ffb640 1 -- 172.21.15.57:0/3834999762 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300c0ab1b0 con 0x7f300c0af8e0 2024-03-29T10:43:34.020 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.035+0000 7f3009ffb640 10 monclient: _finish_auth 0 2024-03-29T10:43:34.020 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.035+0000 7f3009ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:43:34.020 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.035+0000 7f3009ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:34.020 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.035+0000 7f3009ffb640 1 -- 172.21.15.57:0/3834999762 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f3004001990 con 0x7f300c0af8e0 2024-03-29T10:43:34.020 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.035+0000 7f3009ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:34.020 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.035+0000 7f3009ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:34.020 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.035+0000 7f3009ffb640 10 monclient: dump: 2024-03-29T10:43:34.020 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:34.020 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:34.020 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:34.020 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:34.020 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:34.021 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:34.021 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:34.021 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:34.021 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:34.021 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:34.021 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.035+0000 7f301158e640 5 monclient: authenticate success, global_id 4379 2024-03-29T10:43:34.021 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.035+0000 7f301158e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:34.022 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.036+0000 7f3009ffb640 1 -- 172.21.15.57:0/3834999762 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3004001bb0 con 0x7f300c0af8e0 2024-03-29T10:43:34.022 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.036+0000 7f3009ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:34.023 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.036+0000 7f3009ffb640 1 -- 172.21.15.57:0/3834999762 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f3004001f40 con 0x7f300c0af8e0 2024-03-29T10:43:34.023 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.036+0000 7f3009ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:34.023 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.036+0000 7f3009ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:34.023 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.036+0000 7f3009ffb640 10 monclient: dump: 2024-03-29T10:43:34.023 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:34.023 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:34.023 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:34.023 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:34.023 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:34.023 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:34.023 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:34.023 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:34.024 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:34.024 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:34.024 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.038+0000 7f301158e640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:34.024 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.038+0000 7f301158e640 4 set_mon_vals no callback set 2024-03-29T10:43:34.024 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.039+0000 7f301158e640 10 monclient: shutdown 2024-03-29T10:43:34.024 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.039+0000 7f301158e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:34.024 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.039+0000 7f301158e640 1 -- 172.21.15.57:0/3834999762 >> v1:172.21.15.57:6790/0 conn(0x7f300c0af8e0 legacy=0x7f300c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:34.024 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.039+0000 7f301158e640 1 -- 172.21.15.57:0/3834999762 shutdown_connections 2024-03-29T10:43:34.024 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.039+0000 7f301158e640 1 -- 172.21.15.57:0/3834999762 >> 172.21.15.57:0/3834999762 conn(0x7f300c06b660 msgr2=0x7f300c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:34.025 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.042+0000 7f301158e640 1 -- 172.21.15.57:0/3834999762 shutdown_connections 2024-03-29T10:43:34.025 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.042+0000 7f301158e640 1 -- 172.21.15.57:0/3834999762 wait complete. 2024-03-29T10:43:34.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.043+0000 7f301158e640 10 monclient: build_initial_monmap 2024-03-29T10:43:34.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.043+0000 7f301158e640 10 monclient: monmap: 2024-03-29T10:43:34.026 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:43:34.026 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:34.026 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:43:34.026 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:43:34.027 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:34.027 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:34.027 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:43:34.027 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:43:34.027 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:43:34.027 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:34.028 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.044+0000 7f301158e640 1 librados: starting msgr at 2024-03-29T10:43:34.028 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.044+0000 7f301158e640 1 librados: starting objecter 2024-03-29T10:43:34.028 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.044+0000 7f301158e640 1 Processor -- start 2024-03-29T10:43:34.028 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.044+0000 7f301158e640 1 -- start start 2024-03-29T10:43:34.028 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.044+0000 7f301158e640 1 librados: setting wanted keys 2024-03-29T10:43:34.029 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.044+0000 7f301158e640 1 librados: calling monclient init 2024-03-29T10:43:34.029 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.044+0000 7f301158e640 10 monclient: init 2024-03-29T10:43:34.029 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.045+0000 7f301158e640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:34.029 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.045+0000 7f301158e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:43:34.029 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.045+0000 7f301158e640 10 monclient(hunting): picked mon.noname-a con 0x7f300c0af8e0 addr v1:172.21.15.57:6789/0 2024-03-29T10:43:34.029 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.045+0000 7f301158e640 10 monclient(hunting): picked mon.noname-c con 0x7f300c0b0860 addr v1:172.21.15.57:6790/0 2024-03-29T10:43:34.031 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.048+0000 7f301158e640 10 monclient(hunting): picked mon.noname-b con 0x7f300c13ef30 addr v1:172.21.15.179:6789/0 2024-03-29T10:43:34.032 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.048+0000 7f301158e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f300c0b3440 con 0x7f300c0af8e0 2024-03-29T10:43:34.032 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.048+0000 7f300b7fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f300c13ef30 0x7f300c1de480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:51121/0 (socket says 172.21.15.57:51121) 2024-03-29T10:43:34.032 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.048+0000 7f300b7fe640 1 -- 172.21.15.57:0/1490784421 learned_addr learned my addr 172.21.15.57:0/1490784421 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:43:34.032 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.049+0000 7f301158e640 1 -- 172.21.15.57:0/1490784421 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f300c0b2430 con 0x7f300c13ef30 2024-03-29T10:43:34.032 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.049+0000 7f301158e640 1 -- 172.21.15.57:0/1490784421 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f300c1deb90 con 0x7f300c0b0860 2024-03-29T10:43:34.032 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.049+0000 7f301158e640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:34.032 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.049+0000 7f301158e640 10 monclient(hunting): authenticate will time out at 1166.225253s 2024-03-29T10:43:34.039 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.051+0000 7f2ff37fe640 10 client.?.objecter ms_handle_connect 0x7f300c13ef30 2024-03-29T10:43:34.039 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.051+0000 7f2ff37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:34.039 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.051+0000 7f2ff37fe640 10 client.?.objecter ms_handle_connect 0x7f300c0af8e0 2024-03-29T10:43:34.039 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.051+0000 7f2ff37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:34.039 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.051+0000 7f2ff37fe640 1 -- 172.21.15.57:0/1490784421 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3883776028 0 0) 0x7f300c0b2430 con 0x7f300c13ef30 2024-03-29T10:43:34.039 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.051+0000 7f2ff37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:34.039 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.051+0000 7f2ff37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:34.039 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.052+0000 7f2ff37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:34.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.052+0000 7f2ff37fe640 10 monclient(hunting): my global_id is 4366 2024-03-29T10:43:34.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.052+0000 7f2ff37fe640 1 -- 172.21.15.57:0/1490784421 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ff8004eb0 con 0x7f300c13ef30 2024-03-29T10:43:34.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.052+0000 7f2ff37fe640 1 -- 172.21.15.57:0/1490784421 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1457170784 0 0) 0x7f300c0b3440 con 0x7f300c0af8e0 2024-03-29T10:43:34.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.052+0000 7f2ff37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:34.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.052+0000 7f2ff37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:34.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.052+0000 7f2ff37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:34.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.052+0000 7f2ff37fe640 10 monclient(hunting): my global_id is 4353 2024-03-29T10:43:34.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.052+0000 7f2ff37fe640 1 -- 172.21.15.57:0/1490784421 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f300c0b2430 con 0x7f300c0af8e0 2024-03-29T10:43:34.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.053+0000 7f2ff37fe640 1 -- 172.21.15.57:0/1490784421 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3452822409 0 0) 0x7f300c0b2430 con 0x7f300c0af8e0 2024-03-29T10:43:34.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.053+0000 7f2ff37fe640 1 -- 172.21.15.57:0/1490784421 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f300c0b3440 con 0x7f300c0af8e0 2024-03-29T10:43:34.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.053+0000 7f2ff37fe640 1 -- 172.21.15.57:0/1490784421 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f3000001830 con 0x7f300c0af8e0 2024-03-29T10:43:34.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.053+0000 7f2ff37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:34.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.053+0000 7f2ff37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:34.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.053+0000 7f2ff37fe640 10 monclient(hunting): dump: 2024-03-29T10:43:34.041 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:34.041 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:34.041 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:34.041 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:34.041 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:34.041 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:34.041 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:34.041 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:34.041 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:34.041 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:34.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.053+0000 7f2ff37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:34.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.053+0000 7f2ff37fe640 10 client.?.objecter ms_handle_connect 0x7f300c0b0860 2024-03-29T10:43:34.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.053+0000 7f2ff37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:34.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.053+0000 7f2ff37fe640 1 -- 172.21.15.57:0/1490784421 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 791092638 0 0) 0x7f300c0b3440 con 0x7f300c0af8e0 2024-03-29T10:43:34.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.053+0000 7f2ff37fe640 1 -- 172.21.15.57:0/1490784421 >> v1:172.21.15.57:6790/0 conn(0x7f300c0b0860 legacy=0x7f300c1ddd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:34.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.053+0000 7f2ff37fe640 1 -- 172.21.15.57:0/1490784421 >> v1:172.21.15.179:6789/0 conn(0x7f300c13ef30 legacy=0x7f300c1de480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:34.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.053+0000 7f2ff37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:43:34.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.053+0000 7f2ff37fe640 1 monclient: found mon.a 2024-03-29T10:43:34.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.053+0000 7f2ff37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:34.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.053+0000 7f2ff37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:34.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.053+0000 7f2ff37fe640 1 -- 172.21.15.57:0/1490784421 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300c135580 con 0x7f300c0af8e0 2024-03-29T10:43:34.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.053+0000 7f2ff37fe640 10 monclient: _finish_auth 0 2024-03-29T10:43:34.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.053+0000 7f2ff37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:43:34.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.053+0000 7f2ff37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:34.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.054+0000 7f2ff37fe640 1 -- 172.21.15.57:0/1490784421 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30000035f0 con 0x7f300c0af8e0 2024-03-29T10:43:34.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.054+0000 7f2ff37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:34.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.054+0000 7f30097fa640 4 set_mon_vals no callback set 2024-03-29T10:43:34.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.055+0000 7f2ff37fe640 1 -- 172.21.15.57:0/1490784421 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f3000003900 con 0x7f300c0af8e0 2024-03-29T10:43:34.043 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.055+0000 7f2ff37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:34.043 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.055+0000 7f2ff37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:34.043 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.055+0000 7f2ff37fe640 10 monclient: dump: 2024-03-29T10:43:34.043 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:34.043 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:34.043 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:34.043 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:34.043 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:34.043 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:34.043 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:34.043 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:34.043 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:34.043 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:34.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.055+0000 7f301158e640 5 monclient: authenticate success, global_id 4353 2024-03-29T10:43:34.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.055+0000 7f301158e640 10 monclient: _renew_subs 2024-03-29T10:43:34.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.055+0000 7f301158e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:34.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.055+0000 7f301158e640 1 -- 172.21.15.57:0/1490784421 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f300c0b26a0 con 0x7f300c0af8e0 2024-03-29T10:43:34.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.055+0000 7f301158e640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:34.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.055+0000 7f301158e640 10 monclient: _renew_subs 2024-03-29T10:43:34.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.055+0000 7f301158e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:34.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.055+0000 7f301158e640 1 -- 172.21.15.57:0/1490784421 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f300c138490 con 0x7f300c0af8e0 2024-03-29T10:43:34.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.055+0000 7f301158e640 1 librados: init done 2024-03-29T10:43:34.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.057+0000 7f301158e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:34.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.057+0000 7f301158e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:34.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.057+0000 7f301158e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:34.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.057+0000 7f301158e640 1 -- 172.21.15.57:0/1490784421 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f300c137c20 con 0x7f300c0af8e0 2024-03-29T10:43:34.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.061+0000 7f2ff37fe640 1 -- 172.21.15.57:0/1490784421 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f30000198d0 con 0x7f300c0af8e0 2024-03-29T10:43:34.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.062+0000 7f2ff37fe640 1 -- 172.21.15.57:0/1490784421 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f3000083800 con 0x7f300c0af8e0 2024-03-29T10:43:34.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.062+0000 7f2ff37fe640 10 client.4353.objecter ms_dispatch 0x7f300c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:34.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.062+0000 7f2ff37fe640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:34.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.062+0000 7f2ff37fe640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:34.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.062+0000 7f2ff37fe640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-29T10:43:34.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.064+0000 7f300a7fc640 10 monclient: get_auth_request con 0x7f2ff806de20 auth_method 0 2024-03-29T10:43:34.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.066+0000 7f2ff37fe640 10 client.4353.objecter ms_handle_connect 0x7f2ff806de20 2024-03-29T10:43:34.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.070+0000 7f2ff37fe640 1 -- 172.21.15.57:0/1490784421 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f300004ef10 con 0x7f300c0af8e0 2024-03-29T10:43:34.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.071+0000 7f2ff37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:34.054 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.071+0000 7f2ff37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:34.382 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.399+0000 7f301158e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:34.382 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.399+0000 7f301158e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:34.383 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.399+0000 7f301158e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:34.383 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.399+0000 7f301158e640 1 -- 172.21.15.57:0/1490784421 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f300c075f10 con 0x7f300c0af8e0 2024-03-29T10:43:34.383 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.400+0000 7f2ff37fe640 1 -- 172.21.15.57:0/1490784421 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f3000051010 con 0x7f300c0af8e0 2024-03-29T10:43:34.383 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.400+0000 7f2ff37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:34.383 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.400+0000 7f2ff37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:34.384 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:43:34.385 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:34.386 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.404+0000 7f301158e640 10 librados: watch_flush enter 2024-03-29T10:43:34.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.405+0000 7f301158e640 10 librados: watch_flush exit 2024-03-29T10:43:34.388 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.405+0000 7f301158e640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-29T10:43:34.388 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.405+0000 7f301158e640 10 client.4353.objecter successfully canceled tick 2024-03-29T10:43:34.388 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.405+0000 7f301158e640 1 -- 172.21.15.57:0/1490784421 >> v1:172.21.15.179:6816/2698295936 conn(0x7f2ff806de20 legacy=0x7f2ff80702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:34.388 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.405+0000 7f301158e640 10 monclient: shutdown 2024-03-29T10:43:34.388 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.405+0000 7f301158e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:34.388 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.405+0000 7f301158e640 1 -- 172.21.15.57:0/1490784421 >> v1:172.21.15.57:6789/0 conn(0x7f300c0af8e0 legacy=0x7f300c0b38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:34.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.406+0000 7f301158e640 1 -- 172.21.15.57:0/1490784421 shutdown_connections 2024-03-29T10:43:34.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.406+0000 7f301158e640 1 -- 172.21.15.57:0/1490784421 >> 172.21.15.57:0/1490784421 conn(0x7f300c06b660 msgr2=0x7f300c13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:34.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.406+0000 7f301158e640 1 -- 172.21.15.57:0/1490784421 shutdown_connections 2024-03-29T10:43:34.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.406+0000 7f301158e640 1 -- 172.21.15.57:0/1490784421 wait complete. 2024-03-29T10:43:34.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:34.407+0000 7f301158e640 1 librados: shutdown 2024-03-29T10:43:34.406 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:41.213 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:41.369 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.387+0000 7fdf12ed0640 10 monclient: get_monmap_and_config 2024-03-29T10:43:41.370 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.387+0000 7fdf12ed0640 10 monclient: build_initial_monmap 2024-03-29T10:43:41.370 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.387+0000 7fdf12ed0640 10 monclient: monmap: 2024-03-29T10:43:41.370 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:43:41.370 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:41.370 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:43:41.387779+0000 2024-03-29T10:43:41.370 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:43:41.387779+0000 2024-03-29T10:43:41.370 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:41.370 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:41.370 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:43:41.370 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:43:41.370 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:43:41.371 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:41.372 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.389+0000 7fdf12ed0640 1 Processor -- start 2024-03-29T10:43:41.372 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.389+0000 7fdf12ed0640 1 -- start start 2024-03-29T10:43:41.372 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.390+0000 7fdf12ed0640 10 monclient: init 2024-03-29T10:43:41.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.390+0000 7fdf12ed0640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:41.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.390+0000 7fdf12ed0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:43:41.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.390+0000 7fdf12ed0640 10 monclient(hunting): picked mon.noname-c con 0x7fdf04006d80 addr v1:172.21.15.179:6789/0 2024-03-29T10:43:41.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.390+0000 7fdf12ed0640 10 monclient(hunting): picked mon.noname-a con 0x7fdf040cc310 addr v1:172.21.15.57:6789/0 2024-03-29T10:43:41.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.390+0000 7fdf12ed0640 10 monclient(hunting): picked mon.noname-b con 0x7fdf040c9430 addr v1:172.21.15.57:6790/0 2024-03-29T10:43:41.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.390+0000 7fdf12ed0640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf04008470 con 0x7fdf040cc310 2024-03-29T10:43:41.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.390+0000 7fdf12ed0640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf0402be30 con 0x7fdf04006d80 2024-03-29T10:43:41.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.390+0000 7fdf12ed0640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf0402c110 con 0x7fdf040c9430 2024-03-29T10:43:41.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.391+0000 7fdf12ed0640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:41.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.391+0000 7fdf12ed0640 10 monclient(hunting): authenticate will time out at 1173.567039s 2024-03-29T10:43:41.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.395+0000 7fdf116cd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fdf040cc310 0x7fdf040c8d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46187/0 (socket says 172.21.15.57:46187) 2024-03-29T10:43:41.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.395+0000 7fdf116cd640 1 -- 172.21.15.57:0/3771135981 learned_addr learned my addr 172.21.15.57:0/3771135981 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:43:41.379 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.395+0000 7fdf10ecc640 1 -- 172.21.15.57:0/3771135981 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345104032 0 0) 0x7fdf04008470 con 0x7fdf040cc310 2024-03-29T10:43:41.379 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.395+0000 7fdf10ecc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:41.379 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.395+0000 7fdf10ecc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:41.379 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.395+0000 7fdf10ecc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:41.379 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.395+0000 7fdf10ecc640 10 monclient(hunting): my global_id is 4356 2024-03-29T10:43:41.379 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.395+0000 7fdf10ecc640 1 -- 172.21.15.57:0/3771135981 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdef8003520 con 0x7fdf040cc310 2024-03-29T10:43:41.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf10ecc640 1 -- 172.21.15.57:0/3771135981 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1217554337 0 0) 0x7fdf0402c110 con 0x7fdf040c9430 2024-03-29T10:43:41.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf10ecc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:41.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf10ecc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:41.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf10ecc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:41.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf10ecc640 10 monclient(hunting): my global_id is 4382 2024-03-29T10:43:41.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf10ecc640 1 -- 172.21.15.57:0/3771135981 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf04008470 con 0x7fdf040c9430 2024-03-29T10:43:41.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf10ecc640 1 -- 172.21.15.57:0/3771135981 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4158875283 0 0) 0x7fdef8003520 con 0x7fdf040cc310 2024-03-29T10:43:41.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf10ecc640 1 -- 172.21.15.57:0/3771135981 >> v1:172.21.15.57:6790/0 conn(0x7fdf040c9430 legacy=0x7fdf040d74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:41.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf10ecc640 1 -- 172.21.15.57:0/3771135981 >> v1:172.21.15.179:6789/0 conn(0x7fdf04006d80 legacy=0x7fdf040cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:41.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf10ecc640 10 monclient: _finish_hunting 0 2024-03-29T10:43:41.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf10ecc640 1 monclient: found mon.noname-a 2024-03-29T10:43:41.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf10ecc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:41.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf10ecc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:43:41.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf10ecc640 1 -- 172.21.15.57:0/3771135981 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0402c3f0 con 0x7fdf040cc310 2024-03-29T10:43:41.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf10ecc640 10 monclient: _finish_auth 0 2024-03-29T10:43:41.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf10ecc640 10 monclient: _check_auth_tickets 2024-03-29T10:43:41.381 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf10ecc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:41.381 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf10ecc640 1 -- 172.21.15.57:0/3771135981 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fdefc001690 con 0x7fdf040cc310 2024-03-29T10:43:41.381 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf10ecc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:41.381 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf10ecc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:41.381 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf10ecc640 10 monclient: dump: 2024-03-29T10:43:41.381 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:41.381 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:41.381 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:41.381 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:41.381 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:41.381 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:41.381 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:41.381 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:41.381 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:41.381 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:41.382 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf12ed0640 5 monclient: authenticate success, global_id 4356 2024-03-29T10:43:41.382 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf12ed0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:41.383 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf10ecc640 1 -- 172.21.15.57:0/3771135981 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdefc0018b0 con 0x7fdf040cc310 2024-03-29T10:43:41.383 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf10ecc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:41.383 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf12ed0640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:41.383 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.396+0000 7fdf12ed0640 4 set_mon_vals no callback set 2024-03-29T10:43:41.383 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.397+0000 7fdf12ed0640 10 monclient: shutdown 2024-03-29T10:43:41.383 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.397+0000 7fdf12ed0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:41.383 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.397+0000 7fdf12ed0640 1 -- 172.21.15.57:0/3771135981 >> v1:172.21.15.57:6789/0 conn(0x7fdf040cc310 legacy=0x7fdf040c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:41.384 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.397+0000 7fdf10ecc640 1 -- 172.21.15.57:0/3771135981 <== mon.0 v1:172.21.15.57:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fdefc001690 con 0x7fdf040cc310 2024-03-29T10:43:41.384 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.397+0000 7fdf10ecc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:43:41.384 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.397+0000 7fdf12ed0640 1 -- 172.21.15.57:0/3771135981 shutdown_connections 2024-03-29T10:43:41.384 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.397+0000 7fdf12ed0640 1 -- 172.21.15.57:0/3771135981 >> 172.21.15.57:0/3771135981 conn(0x7fdf04005c00 msgr2=0x7fdf04006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:41.384 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.401+0000 7fdf12ed0640 1 -- 172.21.15.57:0/3771135981 shutdown_connections 2024-03-29T10:43:41.384 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.401+0000 7fdf12ed0640 1 -- 172.21.15.57:0/3771135981 wait complete. 2024-03-29T10:43:41.385 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.402+0000 7fdf12ed0640 10 monclient: build_initial_monmap 2024-03-29T10:43:41.385 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.402+0000 7fdf12ed0640 10 monclient: monmap: 2024-03-29T10:43:41.385 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:43:41.385 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:41.385 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:43:41.385 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:43:41.385 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:41.385 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:41.385 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:43:41.385 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:43:41.385 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:43:41.386 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:41.386 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.402+0000 7fdf12ed0640 1 librados: starting msgr at 2024-03-29T10:43:41.386 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.402+0000 7fdf12ed0640 1 librados: starting objecter 2024-03-29T10:43:41.386 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.402+0000 7fdf12ed0640 1 Processor -- start 2024-03-29T10:43:41.386 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.403+0000 7fdf12ed0640 1 -- start start 2024-03-29T10:43:41.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.403+0000 7fdf12ed0640 1 librados: setting wanted keys 2024-03-29T10:43:41.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.403+0000 7fdf12ed0640 1 librados: calling monclient init 2024-03-29T10:43:41.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.403+0000 7fdf12ed0640 10 monclient: init 2024-03-29T10:43:41.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.403+0000 7fdf12ed0640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:41.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.403+0000 7fdf12ed0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:43:41.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.403+0000 7fdf12ed0640 10 monclient(hunting): picked mon.noname-c con 0x7fdf040c9430 addr v1:172.21.15.57:6790/0 2024-03-29T10:43:41.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.403+0000 7fdf12ed0640 10 monclient(hunting): picked mon.noname-b con 0x7fdf04033ff0 addr v1:172.21.15.179:6789/0 2024-03-29T10:43:41.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.403+0000 7fdf12ed0640 10 monclient(hunting): picked mon.noname-a con 0x7fdf0402cfc0 addr v1:172.21.15.57:6789/0 2024-03-29T10:43:41.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.403+0000 7fdf12ed0640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf041650b0 con 0x7fdf0402cfc0 2024-03-29T10:43:41.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.403+0000 7fdf12ed0640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf040c6270 con 0x7fdf04033ff0 2024-03-29T10:43:41.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.403+0000 7fdf12ed0640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf04164b90 con 0x7fdf040c9430 2024-03-29T10:43:41.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.403+0000 7fdf12ed0640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:41.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.403+0000 7fdf12ed0640 10 monclient(hunting): authenticate will time out at 1173.579891s 2024-03-29T10:43:41.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.408+0000 7fdf126cf640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fdf0402cfc0 0x7fdf040c5b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46441/0 (socket says 172.21.15.57:46441) 2024-03-29T10:43:41.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.408+0000 7fdf126cf640 1 -- 172.21.15.57:0/1879268424 learned_addr learned my addr 172.21.15.57:0/1879268424 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:43:41.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.408+0000 7fdf027fc640 10 client.?.objecter ms_handle_connect 0x7fdf0402cfc0 2024-03-29T10:43:41.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.408+0000 7fdf027fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:41.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.408+0000 7fdf027fc640 1 -- 172.21.15.57:0/1879268424 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488346593 0 0) 0x7fdf041650b0 con 0x7fdf0402cfc0 2024-03-29T10:43:41.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.408+0000 7fdf027fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:41.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.408+0000 7fdf027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:41.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.408+0000 7fdf027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:41.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.408+0000 7fdf027fc640 10 monclient(hunting): my global_id is 4359 2024-03-29T10:43:41.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.408+0000 7fdf027fc640 1 -- 172.21.15.57:0/1879268424 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdef0004eb0 con 0x7fdf0402cfc0 2024-03-29T10:43:41.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.409+0000 7fdf027fc640 1 -- 172.21.15.57:0/1879268424 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2542975622 0 0) 0x7fdef0004eb0 con 0x7fdf0402cfc0 2024-03-29T10:43:41.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.409+0000 7fdf027fc640 1 -- 172.21.15.57:0/1879268424 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdef0005300 con 0x7fdf0402cfc0 2024-03-29T10:43:41.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.409+0000 7fdf027fc640 1 -- 172.21.15.57:0/1879268424 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fdf0c04fdd0 con 0x7fdf0402cfc0 2024-03-29T10:43:41.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.409+0000 7fdf027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:41.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.409+0000 7fdf027fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:41.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.409+0000 7fdf027fc640 10 monclient(hunting): dump: 2024-03-29T10:43:41.394 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:41.395 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:41.395 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:41.395 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:41.395 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:41.395 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:41.395 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:41.395 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:41.395 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:41.395 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:41.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.409+0000 7fdf027fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:41.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.410+0000 7fdf027fc640 1 -- 172.21.15.57:0/1879268424 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1478888703 0 0) 0x7fdef0005300 con 0x7fdf0402cfc0 2024-03-29T10:43:41.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.410+0000 7fdf027fc640 1 -- 172.21.15.57:0/1879268424 >> v1:172.21.15.57:6790/0 conn(0x7fdf040c9430 legacy=0x7fdf0402c8b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:43:41.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.412+0000 7fdf027fc640 1 -- 172.21.15.57:0/1879268424 >> v1:172.21.15.179:6789/0 conn(0x7fdf04033ff0 legacy=0x7fdf040c5450 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:43:41.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.412+0000 7fdf027fc640 10 monclient: _finish_hunting 0 2024-03-29T10:43:41.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.412+0000 7fdf027fc640 1 monclient: found mon.a 2024-03-29T10:43:41.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.412+0000 7fdf027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:41.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.412+0000 7fdf027fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:41.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.412+0000 7fdf027fc640 1 -- 172.21.15.57:0/1879268424 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf040cbd70 con 0x7fdf0402cfc0 2024-03-29T10:43:41.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.412+0000 7fdf027fc640 10 monclient: _finish_auth 0 2024-03-29T10:43:41.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.412+0000 7fdf027fc640 10 monclient: _check_auth_tickets 2024-03-29T10:43:41.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.412+0000 7fdf027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:41.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.412+0000 7fdf12ed0640 5 monclient: authenticate success, global_id 4359 2024-03-29T10:43:41.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.412+0000 7fdf12ed0640 10 monclient: _renew_subs 2024-03-29T10:43:41.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.412+0000 7fdf12ed0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:41.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.412+0000 7fdf12ed0640 1 -- 172.21.15.57:0/1879268424 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf040cbfc0 con 0x7fdf0402cfc0 2024-03-29T10:43:41.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.412+0000 7fdf12ed0640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:41.401 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.412+0000 7fdf12ed0640 10 monclient: _renew_subs 2024-03-29T10:43:41.401 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.412+0000 7fdf12ed0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:41.401 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.412+0000 7fdf12ed0640 1 -- 172.21.15.57:0/1879268424 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf04164600 con 0x7fdf0402cfc0 2024-03-29T10:43:41.401 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.412+0000 7fdf12ed0640 1 librados: init done 2024-03-29T10:43:41.401 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.415+0000 7fdf027fc640 1 -- 172.21.15.57:0/1879268424 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf0c04f400 con 0x7fdf0402cfc0 2024-03-29T10:43:41.401 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.415+0000 7fdf027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:41.401 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.415+0000 7fdf027fc640 1 -- 172.21.15.57:0/1879268424 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fdf0c075710 con 0x7fdf0402cfc0 2024-03-29T10:43:41.401 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.415+0000 7fdf027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:41.401 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.415+0000 7fdef7fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:41.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.415+0000 7fdf027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:41.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.415+0000 7fdf027fc640 10 monclient: dump: 2024-03-29T10:43:41.402 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:41.402 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:41.402 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:41.402 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:41.402 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:41.402 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:41.403 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:41.403 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:41.403 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:41.403 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:41.403 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.415+0000 7fdf03fff640 4 set_mon_vals no callback set 2024-03-29T10:43:41.403 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.416+0000 7fdf027fc640 1 -- 172.21.15.57:0/1879268424 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fdf0c067bf0 con 0x7fdf0402cfc0 2024-03-29T10:43:41.404 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.419+0000 7fdf027fc640 1 -- 172.21.15.57:0/1879268424 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7fdf0c0ff6d0 con 0x7fdf0402cfc0 2024-03-29T10:43:41.404 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.419+0000 7fdf027fc640 10 client.4359.objecter ms_dispatch 0x7fdf0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:41.404 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.419+0000 7fdf027fc640 3 client.4359.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:41.404 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.419+0000 7fdf027fc640 3 client.4359.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:41.405 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.419+0000 7fdf027fc640 20 client.4359.objecter dump_active .. 0 homeless 2024-03-29T10:43:41.405 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.419+0000 7fdef7fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:41.405 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.420+0000 7fdef7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:41.405 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.420+0000 7fdef7fff640 1 -- 172.21.15.57:0/1879268424 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf04164d20 con 0x7fdf0402cfc0 2024-03-29T10:43:41.405 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.420+0000 7fdf11ece640 10 monclient: get_auth_request con 0x7fdef006dde0 auth_method 0 2024-03-29T10:43:41.405 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.420+0000 7fdf027fc640 10 client.4359.objecter ms_handle_connect 0x7fdef006dde0 2024-03-29T10:43:41.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.430+0000 7fdf027fc640 1 -- 172.21.15.57:0/1879268424 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdf0c0cafb0 con 0x7fdf0402cfc0 2024-03-29T10:43:41.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.430+0000 7fdf027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:41.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.430+0000 7fdf027fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:41.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.776+0000 7fdef7fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:41.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.776+0000 7fdef7fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:41.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.776+0000 7fdef7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:41.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.776+0000 7fdef7fff640 1 -- 172.21.15.57:0/1879268424 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf040cc150 con 0x7fdf0402cfc0 2024-03-29T10:43:41.760 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:43:41.760 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:41.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.777+0000 7fdf027fc640 1 -- 172.21.15.57:0/1879268424 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7fdf0c0cafb0 con 0x7fdf0402cfc0 2024-03-29T10:43:41.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.777+0000 7fdf027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:41.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.777+0000 7fdf027fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:41.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.781+0000 7fdef7fff640 10 librados: watch_flush enter 2024-03-29T10:43:41.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.781+0000 7fdef7fff640 10 librados: watch_flush exit 2024-03-29T10:43:41.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.781+0000 7fdef7fff640 20 client.4359.objecter shutdown clearing up homeless session... 2024-03-29T10:43:41.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.781+0000 7fdef7fff640 10 client.4359.objecter successfully canceled tick 2024-03-29T10:43:41.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.781+0000 7fdef7fff640 1 -- 172.21.15.57:0/1879268424 >> v1:172.21.15.179:6816/2698295936 conn(0x7fdef006dde0 legacy=0x7fdef00702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:41.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.781+0000 7fdef7fff640 10 monclient: shutdown 2024-03-29T10:43:41.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.781+0000 7fdef7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:41.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.781+0000 7fdef7fff640 1 -- 172.21.15.57:0/1879268424 >> v1:172.21.15.57:6789/0 conn(0x7fdf0402cfc0 legacy=0x7fdf040c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:41.766 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.781+0000 7fdef7fff640 1 -- 172.21.15.57:0/1879268424 shutdown_connections 2024-03-29T10:43:41.766 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.781+0000 7fdef7fff640 1 -- 172.21.15.57:0/1879268424 >> 172.21.15.57:0/1879268424 conn(0x7fdf04005c00 msgr2=0x7fdf04004890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:41.766 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.782+0000 7fdef7fff640 1 -- 172.21.15.57:0/1879268424 shutdown_connections 2024-03-29T10:43:41.766 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.783+0000 7fdef7fff640 1 -- 172.21.15.57:0/1879268424 wait complete. 2024-03-29T10:43:41.766 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:41.783+0000 7fdef7fff640 1 librados: shutdown 2024-03-29T10:43:41.784 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:48.592 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:48.746 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.762+0000 7f2cdf44a640 10 monclient: get_monmap_and_config 2024-03-29T10:43:48.746 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.762+0000 7f2cdf44a640 10 monclient: build_initial_monmap 2024-03-29T10:43:48.746 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.763+0000 7f2cdf44a640 10 monclient: monmap: 2024-03-29T10:43:48.746 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:43:48.746 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:48.746 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:43:48.763702+0000 2024-03-29T10:43:48.746 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:43:48.763702+0000 2024-03-29T10:43:48.746 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:48.747 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:48.747 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:43:48.747 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:43:48.747 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:43:48.747 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:48.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.765+0000 7f2cdf44a640 1 Processor -- start 2024-03-29T10:43:48.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.765+0000 7f2cdf44a640 1 -- start start 2024-03-29T10:43:48.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.765+0000 7f2cdf44a640 10 monclient: init 2024-03-29T10:43:48.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.766+0000 7f2cdf44a640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:48.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.766+0000 7f2cdf44a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:43:48.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.766+0000 7f2cdf44a640 10 monclient(hunting): picked mon.noname-a con 0x7f2cd80b0770 addr v1:172.21.15.57:6789/0 2024-03-29T10:43:48.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.766+0000 7f2cdf44a640 10 monclient(hunting): picked mon.noname-b con 0x7f2cd814a700 addr v1:172.21.15.57:6790/0 2024-03-29T10:43:48.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.769+0000 7f2cdf44a640 10 monclient(hunting): picked mon.noname-c con 0x7f2cd80ac5a0 addr v1:172.21.15.179:6789/0 2024-03-29T10:43:48.753 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.769+0000 7f2cdf44a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd80aba70 con 0x7f2cd80b0770 2024-03-29T10:43:48.753 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.769+0000 7f2cdf44a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd80aecb0 con 0x7f2cd80ac5a0 2024-03-29T10:43:48.754 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.769+0000 7f2cdf44a640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd80af150 con 0x7f2cd814a700 2024-03-29T10:43:48.754 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.769+0000 7f2cdf44a640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:48.754 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.769+0000 7f2cdf44a640 10 monclient(hunting): authenticate will time out at 1180.945786s 2024-03-29T10:43:48.754 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.769+0000 7f2cdd9e7640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f2cd80ac5a0 0x7f2cd8151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:41407/0 (socket says 172.21.15.57:41407) 2024-03-29T10:43:48.754 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.769+0000 7f2cdd9e7640 1 -- 172.21.15.57:0/2640772614 learned_addr learned my addr 172.21.15.57:0/2640772614 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:43:48.754 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.770+0000 7f2ccffff640 1 -- 172.21.15.57:0/2640772614 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3538407131 0 0) 0x7f2cd80aecb0 con 0x7f2cd80ac5a0 2024-03-29T10:43:48.754 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.770+0000 7f2ccffff640 10 monclient(hunting): _negotiate 2024-03-29T10:43:48.754 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.770+0000 7f2ccffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:48.754 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.770+0000 7f2ccffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:48.754 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.770+0000 7f2ccffff640 10 monclient(hunting): my global_id is 4372 2024-03-29T10:43:48.754 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.770+0000 7f2ccffff640 1 -- 172.21.15.57:0/2640772614 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cc4003520 con 0x7f2cd80ac5a0 2024-03-29T10:43:48.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.771+0000 7f2ccffff640 1 -- 172.21.15.57:0/2640772614 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1871095641 0 0) 0x7f2cc4003520 con 0x7f2cd80ac5a0 2024-03-29T10:43:48.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.771+0000 7f2ccffff640 1 -- 172.21.15.57:0/2640772614 >> v1:172.21.15.57:6790/0 conn(0x7f2cd814a700 legacy=0x7f2cd80abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:48.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.771+0000 7f2ccffff640 1 -- 172.21.15.57:0/2640772614 >> v1:172.21.15.57:6789/0 conn(0x7f2cd80b0770 legacy=0x7f2cd8149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:43:48.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.772+0000 7f2ccffff640 10 monclient: _finish_hunting 0 2024-03-29T10:43:48.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.772+0000 7f2ccffff640 1 monclient: found mon.noname-c 2024-03-29T10:43:48.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.772+0000 7f2ccffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:48.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.772+0000 7f2ccffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:43:48.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.772+0000 7f2ccffff640 1 -- 172.21.15.57:0/2640772614 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd80ac9d0 con 0x7f2cd80ac5a0 2024-03-29T10:43:48.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.772+0000 7f2ccffff640 10 monclient: _finish_auth 0 2024-03-29T10:43:48.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.772+0000 7f2ccffff640 10 monclient: _check_auth_tickets 2024-03-29T10:43:48.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.772+0000 7f2ccffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:48.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.772+0000 7f2ccffff640 1 -- 172.21.15.57:0/2640772614 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f2cd40016b0 con 0x7f2cd80ac5a0 2024-03-29T10:43:48.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.772+0000 7f2ccffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:48.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.772+0000 7f2ccffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:48.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.772+0000 7f2ccffff640 10 monclient: dump: 2024-03-29T10:43:48.758 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:48.758 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:48.758 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:48.758 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:48.759 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:48.759 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:48.759 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:48.759 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:48.759 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:48.759 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:48.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.773+0000 7f2cdf44a640 5 monclient: authenticate success, global_id 4372 2024-03-29T10:43:48.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.773+0000 7f2cdf44a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:48.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.773+0000 7f2ccffff640 1 -- 172.21.15.57:0/2640772614 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2cd4001880 con 0x7f2cd80ac5a0 2024-03-29T10:43:48.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.773+0000 7f2ccffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:48.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.773+0000 7f2ccffff640 1 -- 172.21.15.57:0/2640772614 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f2cd4001cd0 con 0x7f2cd80ac5a0 2024-03-29T10:43:48.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.773+0000 7f2ccffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:48.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.773+0000 7f2ccffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:48.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.773+0000 7f2ccffff640 10 monclient: dump: 2024-03-29T10:43:48.760 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:48.760 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:48.760 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:48.760 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:48.760 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:48.760 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:48.760 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:48.760 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:48.760 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:48.760 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:48.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.773+0000 7f2cdf44a640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:48.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.773+0000 7f2cdf44a640 4 set_mon_vals no callback set 2024-03-29T10:43:48.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.774+0000 7f2cdf44a640 10 monclient: shutdown 2024-03-29T10:43:48.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.774+0000 7f2cdf44a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:48.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.774+0000 7f2cdf44a640 1 -- 172.21.15.57:0/2640772614 >> v1:172.21.15.179:6789/0 conn(0x7f2cd80ac5a0 legacy=0x7f2cd8151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:48.761 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.775+0000 7f2cdf44a640 1 -- 172.21.15.57:0/2640772614 shutdown_connections 2024-03-29T10:43:48.761 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.775+0000 7f2cdf44a640 1 -- 172.21.15.57:0/2640772614 >> 172.21.15.57:0/2640772614 conn(0x7f2cd806b660 msgr2=0x7f2cd806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:48.762 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.776+0000 7f2cdf44a640 1 -- 172.21.15.57:0/2640772614 shutdown_connections 2024-03-29T10:43:48.762 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.776+0000 7f2cdf44a640 1 -- 172.21.15.57:0/2640772614 wait complete. 2024-03-29T10:43:48.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.777+0000 7f2cdf44a640 10 monclient: build_initial_monmap 2024-03-29T10:43:48.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.777+0000 7f2cdf44a640 10 monclient: monmap: 2024-03-29T10:43:48.763 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:43:48.763 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:48.763 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:43:48.763 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:43:48.763 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:48.763 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:48.763 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:43:48.763 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:43:48.763 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:43:48.763 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:48.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.778+0000 7f2cdf44a640 1 librados: starting msgr at 2024-03-29T10:43:48.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.778+0000 7f2cdf44a640 1 librados: starting objecter 2024-03-29T10:43:48.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.778+0000 7f2cdf44a640 1 Processor -- start 2024-03-29T10:43:48.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.778+0000 7f2cdf44a640 1 -- start start 2024-03-29T10:43:48.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.778+0000 7f2cdf44a640 1 librados: setting wanted keys 2024-03-29T10:43:48.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.778+0000 7f2cdf44a640 1 librados: calling monclient init 2024-03-29T10:43:48.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.778+0000 7f2cdf44a640 10 monclient: init 2024-03-29T10:43:48.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.779+0000 7f2cdf44a640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:48.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.779+0000 7f2cdf44a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:43:48.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.779+0000 7f2cdf44a640 10 monclient(hunting): picked mon.noname-b con 0x7f2cd80ac5a0 addr v1:172.21.15.179:6789/0 2024-03-29T10:43:48.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.779+0000 7f2cdf44a640 10 monclient(hunting): picked mon.noname-c con 0x7f2cd80b0770 addr v1:172.21.15.57:6790/0 2024-03-29T10:43:48.765 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.782+0000 7f2cdf44a640 10 monclient(hunting): picked mon.noname-a con 0x7f2cd814a700 addr v1:172.21.15.57:6789/0 2024-03-29T10:43:48.765 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.782+0000 7f2cdf44a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd8126590 con 0x7f2cd814a700 2024-03-29T10:43:48.765 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.782+0000 7f2cdf44a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd8128830 con 0x7f2cd80ac5a0 2024-03-29T10:43:48.765 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.782+0000 7f2cdf44a640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd81d9300 con 0x7f2cd80b0770 2024-03-29T10:43:48.765 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.783+0000 7f2cdf44a640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:48.765 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.783+0000 7f2cdf44a640 10 monclient(hunting): authenticate will time out at 1180.959034s 2024-03-29T10:43:48.767 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.784+0000 7f2cdd1e6640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f2cd80ac5a0 0x7f2cd81451b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:41485/0 (socket says 172.21.15.57:41485) 2024-03-29T10:43:48.767 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.784+0000 7f2cdd1e6640 1 -- 172.21.15.57:0/669280858 learned_addr learned my addr 172.21.15.57:0/669280858 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:43:48.769 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.786+0000 7f2ccdffb640 10 client.?.objecter ms_handle_connect 0x7f2cd80ac5a0 2024-03-29T10:43:48.771 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.786+0000 7f2ccdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:48.771 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.786+0000 7f2ccdffb640 1 -- 172.21.15.57:0/669280858 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 874464918 0 0) 0x7f2cd8128830 con 0x7f2cd80ac5a0 2024-03-29T10:43:48.771 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.786+0000 7f2ccdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:43:48.771 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.786+0000 7f2ccdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:48.771 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.786+0000 7f2ccdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:48.771 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.786+0000 7f2ccdffb640 10 monclient(hunting): my global_id is 4375 2024-03-29T10:43:48.771 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.786+0000 7f2ccdffb640 1 -- 172.21.15.57:0/669280858 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cbc004eb0 con 0x7f2cd80ac5a0 2024-03-29T10:43:48.771 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.787+0000 7f2ccdffb640 10 client.?.objecter ms_handle_connect 0x7f2cd814a700 2024-03-29T10:43:48.771 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.787+0000 7f2ccdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:48.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.789+0000 7f2ccdffb640 1 -- 172.21.15.57:0/669280858 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3248447586 0 0) 0x7f2cbc004eb0 con 0x7f2cd80ac5a0 2024-03-29T10:43:48.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 1 -- 172.21.15.57:0/669280858 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cd8128830 con 0x7f2cd80ac5a0 2024-03-29T10:43:48.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 1 -- 172.21.15.57:0/669280858 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878931651 0 0) 0x7f2cd8126590 con 0x7f2cd814a700 2024-03-29T10:43:48.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:43:48.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:48.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:48.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 10 monclient(hunting): my global_id is 4362 2024-03-29T10:43:48.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 1 -- 172.21.15.57:0/669280858 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cbc004eb0 con 0x7f2cd814a700 2024-03-29T10:43:48.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 1 -- 172.21.15.57:0/669280858 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f2cd0001ce0 con 0x7f2cd80ac5a0 2024-03-29T10:43:48.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:48.775 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:48.775 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 10 monclient(hunting): dump: 2024-03-29T10:43:48.775 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:48.775 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:48.775 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:48.775 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:48.775 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:48.775 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:48.775 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:48.775 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:48.775 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:48.776 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:48.776 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:48.776 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 1 -- 172.21.15.57:0/669280858 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 474700664 0 0) 0x7f2cbc004eb0 con 0x7f2cd814a700 2024-03-29T10:43:48.776 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 1 -- 172.21.15.57:0/669280858 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cd8126590 con 0x7f2cd814a700 2024-03-29T10:43:48.776 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 1 -- 172.21.15.57:0/669280858 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f2cd40019f0 con 0x7f2cd814a700 2024-03-29T10:43:48.776 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:48.776 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:48.776 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 10 monclient(hunting): dump: 2024-03-29T10:43:48.776 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:48.776 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:48.776 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:48.776 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:48.776 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:48.776 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:48.777 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:48.777 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:48.777 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:48.777 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:48.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 1 -- 172.21.15.57:0/669280858 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 900895276 0 0) 0x7f2cd8126590 con 0x7f2cd814a700 2024-03-29T10:43:48.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 1 -- 172.21.15.57:0/669280858 >> v1:172.21.15.57:6790/0 conn(0x7f2cd80b0770 legacy=0x7f2cd8126a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:48.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 1 -- 172.21.15.57:0/669280858 >> v1:172.21.15.179:6789/0 conn(0x7f2cd80ac5a0 legacy=0x7f2cd81451b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:48.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 10 monclient: _finish_hunting 0 2024-03-29T10:43:48.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 1 monclient: found mon.a 2024-03-29T10:43:48.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:48.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:48.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.790+0000 7f2ccdffb640 1 -- 172.21.15.57:0/669280858 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd8126160 con 0x7f2cd814a700 2024-03-29T10:43:48.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.791+0000 7f2ccdffb640 10 monclient: _finish_auth 0 2024-03-29T10:43:48.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.791+0000 7f2ccdffb640 10 monclient: _check_auth_tickets 2024-03-29T10:43:48.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.791+0000 7f2ccdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:48.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.791+0000 7f2cdf44a640 5 monclient: authenticate success, global_id 4362 2024-03-29T10:43:48.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.791+0000 7f2cdf44a640 10 monclient: _renew_subs 2024-03-29T10:43:48.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.791+0000 7f2cdf44a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:48.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.791+0000 7f2cdf44a640 1 -- 172.21.15.57:0/669280858 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd81df9f0 con 0x7f2cd814a700 2024-03-29T10:43:48.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.791+0000 7f2cdf44a640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:48.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.791+0000 7f2cdf44a640 10 monclient: _renew_subs 2024-03-29T10:43:48.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.791+0000 7f2cdf44a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:48.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.791+0000 7f2cdf44a640 1 -- 172.21.15.57:0/669280858 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd8145aa0 con 0x7f2cd814a700 2024-03-29T10:43:48.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.791+0000 7f2cdf44a640 1 librados: init done 2024-03-29T10:43:48.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.791+0000 7f2ccdffb640 1 -- 172.21.15.57:0/669280858 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2cd4001250 con 0x7f2cd814a700 2024-03-29T10:43:48.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.791+0000 7f2ccdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:48.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.791+0000 7f2ccdffb640 1 -- 172.21.15.57:0/669280858 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f2cd4004b20 con 0x7f2cd814a700 2024-03-29T10:43:48.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.791+0000 7f2ccdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:48.778 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.791+0000 7f2ccdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:48.779 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.791+0000 7f2ccdffb640 10 monclient: dump: 2024-03-29T10:43:48.779 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:48.779 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:48.779 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:48.779 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:48.779 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:48.779 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:48.779 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:48.779 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:48.779 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:48.779 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:48.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.792+0000 7f2ccf7fe640 4 set_mon_vals no callback set 2024-03-29T10:43:48.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.793+0000 7f2cdf44a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:48.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.793+0000 7f2cdf44a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:48.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.794+0000 7f2cdf44a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:48.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.794+0000 7f2cdf44a640 1 -- 172.21.15.57:0/669280858 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb8005080 con 0x7f2cd814a700 2024-03-29T10:43:48.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.795+0000 7f2ccdffb640 1 -- 172.21.15.57:0/669280858 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f2cd401aaf0 con 0x7f2cd814a700 2024-03-29T10:43:48.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.796+0000 7f2ccdffb640 1 -- 172.21.15.57:0/669280858 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f2cd4084930 con 0x7f2cd814a700 2024-03-29T10:43:48.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.796+0000 7f2ccdffb640 10 client.4362.objecter ms_dispatch 0x7f2cd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:48.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.796+0000 7f2ccdffb640 3 client.4362.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:48.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.796+0000 7f2ccdffb640 3 client.4362.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:48.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.796+0000 7f2ccdffb640 20 client.4362.objecter dump_active .. 0 homeless 2024-03-29T10:43:48.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.796+0000 7f2cdd1e6640 10 monclient: get_auth_request con 0x7f2cbc06dfd0 auth_method 0 2024-03-29T10:43:48.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.797+0000 7f2ccdffb640 10 client.4362.objecter ms_handle_connect 0x7f2cbc06dfd0 2024-03-29T10:43:48.786 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.803+0000 7f2ccdffb640 1 -- 172.21.15.57:0/669280858 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2cd4050210 con 0x7f2cd814a700 2024-03-29T10:43:48.786 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.803+0000 7f2ccdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:48.786 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:48.803+0000 7f2ccdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:49.113 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:49.131+0000 7f2cdf44a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:49.114 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:49.131+0000 7f2cdf44a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:49.114 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:49.131+0000 7f2cdf44a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:43:49.114 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:49.131+0000 7f2cdf44a640 1 -- 172.21.15.57:0/669280858 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2cb8005330 con 0x7f2cd814a700 2024-03-29T10:43:49.115 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:49.132+0000 7f2ccdffb640 1 -- 172.21.15.57:0/669280858 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f2cd4052140 con 0x7f2cd814a700 2024-03-29T10:43:49.115 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:49.132+0000 7f2ccdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:49.115 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:49.132+0000 7f2ccdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:49.117 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:43:49.117 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:49.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:49.135+0000 7f2cdf44a640 10 librados: watch_flush enter 2024-03-29T10:43:49.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:49.136+0000 7f2cdf44a640 10 librados: watch_flush exit 2024-03-29T10:43:49.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:49.136+0000 7f2cdf44a640 20 client.4362.objecter shutdown clearing up homeless session... 2024-03-29T10:43:49.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:49.136+0000 7f2cdf44a640 10 client.4362.objecter successfully canceled tick 2024-03-29T10:43:49.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:49.136+0000 7f2cdf44a640 1 -- 172.21.15.57:0/669280858 >> v1:172.21.15.179:6816/2698295936 conn(0x7f2cbc06dfd0 legacy=0x7f2cbc070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:49.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:49.136+0000 7f2cdf44a640 10 monclient: shutdown 2024-03-29T10:43:49.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:49.136+0000 7f2cdf44a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:49.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:49.136+0000 7f2cdf44a640 1 -- 172.21.15.57:0/669280858 >> v1:172.21.15.57:6789/0 conn(0x7f2cd814a700 legacy=0x7f2cd8127130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:49.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:49.136+0000 7f2cdf44a640 1 -- 172.21.15.57:0/669280858 shutdown_connections 2024-03-29T10:43:49.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:49.136+0000 7f2cdf44a640 1 -- 172.21.15.57:0/669280858 >> 172.21.15.57:0/669280858 conn(0x7f2cd806b660 msgr2=0x7f2cd80b2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:49.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:49.138+0000 7f2cdf44a640 1 -- 172.21.15.57:0/669280858 shutdown_connections 2024-03-29T10:43:49.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:49.138+0000 7f2cdf44a640 1 -- 172.21.15.57:0/669280858 wait complete. 2024-03-29T10:43:49.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:49.138+0000 7f2cdf44a640 1 librados: shutdown 2024-03-29T10:43:49.141 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:55.948 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:56.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.114+0000 7fb5a684e640 10 monclient: get_monmap_and_config 2024-03-29T10:43:56.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.114+0000 7fb5a684e640 10 monclient: build_initial_monmap 2024-03-29T10:43:56.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.114+0000 7fb5a684e640 10 monclient: monmap: 2024-03-29T10:43:56.098 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:43:56.098 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:56.098 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:43:56.114788+0000 2024-03-29T10:43:56.098 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:43:56.114788+0000 2024-03-29T10:43:56.098 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:56.098 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:56.098 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:43:56.098 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:43:56.098 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:43:56.099 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:56.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.116+0000 7fb5a684e640 1 Processor -- start 2024-03-29T10:43:56.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.116+0000 7fb5a684e640 1 -- start start 2024-03-29T10:43:56.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.116+0000 7fb5a684e640 10 monclient: init 2024-03-29T10:43:56.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.116+0000 7fb5a684e640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:56.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.116+0000 7fb5a684e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:43:56.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.116+0000 7fb5a684e640 10 monclient(hunting): picked mon.noname-a con 0x7fb5a00b0860 addr v1:172.21.15.57:6789/0 2024-03-29T10:43:56.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.116+0000 7fb5a684e640 10 monclient(hunting): picked mon.noname-c con 0x7fb5a013ef30 addr v1:172.21.15.179:6789/0 2024-03-29T10:43:56.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.117+0000 7fb5a684e640 10 monclient(hunting): picked mon.noname-b con 0x7fb5a00af8e0 addr v1:172.21.15.57:6790/0 2024-03-29T10:43:56.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.117+0000 7fb5a684e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a00b0c40 con 0x7fb5a00b0860 2024-03-29T10:43:56.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.117+0000 7fb5a684e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a00aed10 con 0x7fb5a013ef30 2024-03-29T10:43:56.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.117+0000 7fb5a684e640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a00afd50 con 0x7fb5a00af8e0 2024-03-29T10:43:56.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.117+0000 7fb5a684e640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:56.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.117+0000 7fb5a684e640 10 monclient(hunting): authenticate will time out at 1188.293071s 2024-03-29T10:43:56.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.119+0000 7fb59ffff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fb5a00b0860 0x7fb5a013e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:54888/0 (socket says 172.21.15.57:54888) 2024-03-29T10:43:56.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.119+0000 7fb59ffff640 1 -- 172.21.15.57:0/2288960113 learned_addr learned my addr 172.21.15.57:0/2288960113 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:43:56.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.121+0000 7fb59effd640 1 -- 172.21.15.57:0/2288960113 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854657175 0 0) 0x7fb5a00b0c40 con 0x7fb5a00b0860 2024-03-29T10:43:56.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.121+0000 7fb59effd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:56.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.121+0000 7fb59effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:56.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.121+0000 7fb59effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:56.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.121+0000 7fb59effd640 10 monclient(hunting): my global_id is 4365 2024-03-29T10:43:56.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.121+0000 7fb59effd640 1 -- 172.21.15.57:0/2288960113 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb588003520 con 0x7fb5a00b0860 2024-03-29T10:43:56.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.121+0000 7fb59effd640 1 -- 172.21.15.57:0/2288960113 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 962316041 0 0) 0x7fb588003520 con 0x7fb5a00b0860 2024-03-29T10:43:56.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.121+0000 7fb59effd640 1 -- 172.21.15.57:0/2288960113 >> v1:172.21.15.57:6790/0 conn(0x7fb5a00af8e0 legacy=0x7fb5a0149a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:43:56.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.122+0000 7fb59effd640 1 -- 172.21.15.57:0/2288960113 >> v1:172.21.15.179:6789/0 conn(0x7fb5a013ef30 legacy=0x7fb5a00af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:43:56.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.123+0000 7fb59effd640 10 monclient: _finish_hunting 0 2024-03-29T10:43:56.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.123+0000 7fb59effd640 1 monclient: found mon.noname-a 2024-03-29T10:43:56.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.123+0000 7fb59effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:56.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.123+0000 7fb59effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:43:56.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.123+0000 7fb59effd640 1 -- 172.21.15.57:0/2288960113 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a00ab1b0 con 0x7fb5a00b0860 2024-03-29T10:43:56.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.123+0000 7fb59effd640 10 monclient: _finish_auth 0 2024-03-29T10:43:56.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.123+0000 7fb59effd640 10 monclient: _check_auth_tickets 2024-03-29T10:43:56.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.123+0000 7fb59effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:56.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.123+0000 7fb5a684e640 5 monclient: authenticate success, global_id 4365 2024-03-29T10:43:56.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.123+0000 7fb5a684e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:56.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.123+0000 7fb59effd640 1 -- 172.21.15.57:0/2288960113 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb598001690 con 0x7fb5a00b0860 2024-03-29T10:43:56.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.123+0000 7fb59effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:56.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.123+0000 7fb59effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:56.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.123+0000 7fb59effd640 10 monclient: dump: 2024-03-29T10:43:56.108 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:56.108 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:56.108 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:56.108 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:56.108 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:56.108 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:56.108 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:56.108 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:56.108 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:56.108 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:56.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.123+0000 7fb5a684e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:56.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.124+0000 7fb59effd640 1 -- 172.21.15.57:0/2288960113 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb598001860 con 0x7fb5a00b0860 2024-03-29T10:43:56.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.124+0000 7fb59effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:56.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.124+0000 7fb5a684e640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:56.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.124+0000 7fb5a684e640 4 set_mon_vals no callback set 2024-03-29T10:43:56.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.125+0000 7fb5a684e640 10 monclient: shutdown 2024-03-29T10:43:56.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.125+0000 7fb5a684e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:56.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.125+0000 7fb5a684e640 1 -- 172.21.15.57:0/2288960113 >> v1:172.21.15.57:6789/0 conn(0x7fb5a00b0860 legacy=0x7fb5a013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:56.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.126+0000 7fb5a684e640 1 -- 172.21.15.57:0/2288960113 shutdown_connections 2024-03-29T10:43:56.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.126+0000 7fb5a684e640 1 -- 172.21.15.57:0/2288960113 >> 172.21.15.57:0/2288960113 conn(0x7fb5a006b660 msgr2=0x7fb5a006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:56.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.126+0000 7fb5a684e640 1 -- 172.21.15.57:0/2288960113 shutdown_connections 2024-03-29T10:43:56.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.126+0000 7fb5a684e640 1 -- 172.21.15.57:0/2288960113 wait complete. 2024-03-29T10:43:56.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.127+0000 7fb5a684e640 10 monclient: build_initial_monmap 2024-03-29T10:43:56.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.127+0000 7fb5a684e640 10 monclient: monmap: 2024-03-29T10:43:56.111 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:43:56.111 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:56.112 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:43:56.112 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:43:56.112 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:56.112 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:56.112 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:43:56.112 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:43:56.112 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:43:56.112 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:56.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.127+0000 7fb5a684e640 1 librados: starting msgr at 2024-03-29T10:43:56.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.127+0000 7fb5a684e640 1 librados: starting objecter 2024-03-29T10:43:56.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.128+0000 7fb5a684e640 1 Processor -- start 2024-03-29T10:43:56.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.128+0000 7fb5a684e640 1 -- start start 2024-03-29T10:43:56.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.128+0000 7fb5a684e640 1 librados: setting wanted keys 2024-03-29T10:43:56.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.128+0000 7fb5a684e640 1 librados: calling monclient init 2024-03-29T10:43:56.113 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.128+0000 7fb5a684e640 10 monclient: init 2024-03-29T10:43:56.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.129+0000 7fb5a684e640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:56.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.129+0000 7fb5a684e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:43:56.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.129+0000 7fb5a684e640 10 monclient(hunting): picked mon.noname-a con 0x7fb5a00af8e0 addr v1:172.21.15.57:6789/0 2024-03-29T10:43:56.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.129+0000 7fb5a684e640 10 monclient(hunting): picked mon.noname-c con 0x7fb5a00b0860 addr v1:172.21.15.57:6790/0 2024-03-29T10:43:56.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.129+0000 7fb5a684e640 10 monclient(hunting): picked mon.noname-b con 0x7fb5a013ef30 addr v1:172.21.15.179:6789/0 2024-03-29T10:43:56.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.129+0000 7fb5a684e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a0070b90 con 0x7fb5a00af8e0 2024-03-29T10:43:56.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.129+0000 7fb5a684e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a0135020 con 0x7fb5a013ef30 2024-03-29T10:43:56.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.129+0000 7fb5a684e640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a006faf0 con 0x7fb5a00b0860 2024-03-29T10:43:56.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.129+0000 7fb5a684e640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:56.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.129+0000 7fb5a684e640 10 monclient(hunting): authenticate will time out at 1188.305595s 2024-03-29T10:43:56.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.129+0000 7fb59ffff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fb5a00af8e0 0x7fb5a0071600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:34594/0 (socket says 172.21.15.57:34594) 2024-03-29T10:43:56.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.130+0000 7fb59ffff640 1 -- 172.21.15.57:0/732001652 learned_addr learned my addr 172.21.15.57:0/732001652 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:43:56.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.130+0000 7fb59cff9640 10 client.?.objecter ms_handle_connect 0x7fb5a00af8e0 2024-03-29T10:43:56.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.130+0000 7fb59cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:56.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.130+0000 7fb59cff9640 1 -- 172.21.15.57:0/732001652 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668088733 0 0) 0x7fb5a0070b90 con 0x7fb5a00af8e0 2024-03-29T10:43:56.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.130+0000 7fb59cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:43:56.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.130+0000 7fb59cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:56.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.130+0000 7fb59cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:56.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.130+0000 7fb59cff9640 10 monclient(hunting): my global_id is 4368 2024-03-29T10:43:56.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.130+0000 7fb59cff9640 1 -- 172.21.15.57:0/732001652 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb58c004eb0 con 0x7fb5a00af8e0 2024-03-29T10:43:56.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.131+0000 7fb59cff9640 10 client.?.objecter ms_handle_connect 0x7fb5a00b0860 2024-03-29T10:43:56.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.131+0000 7fb59cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:56.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.132+0000 7fb59cff9640 1 -- 172.21.15.57:0/732001652 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1166608349 0 0) 0x7fb58c004eb0 con 0x7fb5a00af8e0 2024-03-29T10:43:56.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.132+0000 7fb59cff9640 1 -- 172.21.15.57:0/732001652 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5a0070b90 con 0x7fb5a00af8e0 2024-03-29T10:43:56.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.132+0000 7fb59cff9640 1 -- 172.21.15.57:0/732001652 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb598001d50 con 0x7fb5a00af8e0 2024-03-29T10:43:56.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.132+0000 7fb59cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:56.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.132+0000 7fb59cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:56.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.132+0000 7fb59cff9640 10 monclient(hunting): dump: 2024-03-29T10:43:56.121 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:56.121 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:56.121 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:56.121 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:56.122 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:56.122 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:56.122 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:56.122 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:56.122 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:56.122 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:56.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.132+0000 7fb59cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:56.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.132+0000 7fb59cff9640 1 -- 172.21.15.57:0/732001652 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991978731 0 0) 0x7fb5a006faf0 con 0x7fb5a00b0860 2024-03-29T10:43:56.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.132+0000 7fb59cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:43:56.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.132+0000 7fb59cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:56.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.132+0000 7fb59cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:56.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.132+0000 7fb59cff9640 10 monclient(hunting): my global_id is 4385 2024-03-29T10:43:56.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.132+0000 7fb59cff9640 1 -- 172.21.15.57:0/732001652 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb58c005680 con 0x7fb5a00b0860 2024-03-29T10:43:56.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.133+0000 7fb59cff9640 1 -- 172.21.15.57:0/732001652 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4209721241 0 0) 0x7fb58c005680 con 0x7fb5a00b0860 2024-03-29T10:43:56.123 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.133+0000 7fb59cff9640 1 -- 172.21.15.57:0/732001652 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5a006faf0 con 0x7fb5a00b0860 2024-03-29T10:43:56.123 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.133+0000 7fb59cff9640 1 -- 172.21.15.57:0/732001652 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb594001730 con 0x7fb5a00b0860 2024-03-29T10:43:56.123 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.133+0000 7fb59cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:56.123 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.133+0000 7fb59cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:56.123 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.133+0000 7fb59cff9640 10 monclient(hunting): dump: 2024-03-29T10:43:56.123 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:56.123 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:56.123 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:56.123 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:56.123 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:56.123 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:56.123 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:56.123 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:56.123 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:56.123 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:56.124 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.134+0000 7fb59cff9640 1 -- 172.21.15.57:0/732001652 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1785513933 0 0) 0x7fb5a006faf0 con 0x7fb5a00b0860 2024-03-29T10:43:56.124 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.134+0000 7fb59cff9640 1 -- 172.21.15.57:0/732001652 >> v1:172.21.15.179:6789/0 conn(0x7fb5a013ef30 legacy=0x7fb5a01e00a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:43:56.124 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.134+0000 7fb59cff9640 1 -- 172.21.15.57:0/732001652 >> v1:172.21.15.57:6789/0 conn(0x7fb5a00af8e0 legacy=0x7fb5a0071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:56.124 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.134+0000 7fb59cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:43:56.124 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.134+0000 7fb59cff9640 1 monclient: found mon.c 2024-03-29T10:43:56.124 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.134+0000 7fb59cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:56.124 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.134+0000 7fb59cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:43:56.124 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.134+0000 7fb59cff9640 1 -- 172.21.15.57:0/732001652 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a006fc80 con 0x7fb5a00b0860 2024-03-29T10:43:56.124 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.134+0000 7fb59cff9640 10 monclient: _finish_auth 0 2024-03-29T10:43:56.124 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.134+0000 7fb59cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:43:56.124 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.134+0000 7fb59cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:56.124 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.135+0000 7fb5a684e640 5 monclient: authenticate success, global_id 4385 2024-03-29T10:43:56.124 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.135+0000 7fb5a684e640 10 monclient: _renew_subs 2024-03-29T10:43:56.124 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.135+0000 7fb5a684e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:43:56.124 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.135+0000 7fb5a684e640 1 -- 172.21.15.57:0/732001652 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a006eab0 con 0x7fb5a00b0860 2024-03-29T10:43:56.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.135+0000 7fb59cff9640 1 -- 172.21.15.57:0/732001652 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb594001c20 con 0x7fb5a00b0860 2024-03-29T10:43:56.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.135+0000 7fb59cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:56.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.135+0000 7fb59cff9640 1 -- 172.21.15.57:0/732001652 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb594003790 con 0x7fb5a00b0860 2024-03-29T10:43:56.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.135+0000 7fb59cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:56.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.135+0000 7fb59cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:56.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.135+0000 7fb59cff9640 10 monclient: dump: 2024-03-29T10:43:56.125 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:43:56.125 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:43:56.125 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:56.125 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:43:56.125 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:56.125 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:43:56.125 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:43:56.125 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:43:56.125 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:43:56.126 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:43:56.126 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.135+0000 7fb5a684e640 10 client.4385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:56.126 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.135+0000 7fb5a684e640 10 monclient: _renew_subs 2024-03-29T10:43:56.126 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.135+0000 7fb5a684e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:43:56.126 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.136+0000 7fb5a684e640 1 -- 172.21.15.57:0/732001652 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a006ed60 con 0x7fb5a00b0860 2024-03-29T10:43:56.126 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.136+0000 7fb59e7fc640 4 set_mon_vals no callback set 2024-03-29T10:43:56.126 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.136+0000 7fb5a684e640 1 librados: init done 2024-03-29T10:43:56.127 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.138+0000 7fb59cff9640 1 -- 172.21.15.57:0/732001652 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fb594019760 con 0x7fb5a00b0860 2024-03-29T10:43:56.127 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.139+0000 7fb59cff9640 1 -- 172.21.15.57:0/732001652 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7fb594083680 con 0x7fb5a00b0860 2024-03-29T10:43:56.128 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.139+0000 7fb59cff9640 10 client.4385.objecter ms_dispatch 0x7fb5a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:56.128 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.139+0000 7fb59cff9640 3 client.4385.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:56.128 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.139+0000 7fb59cff9640 3 client.4385.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:56.128 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.139+0000 7fb59cff9640 20 client.4385.objecter dump_active .. 0 homeless 2024-03-29T10:43:56.128 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.139+0000 7fb59ffff640 10 monclient: get_auth_request con 0x7fb58c0725d0 auth_method 0 2024-03-29T10:43:56.128 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.139+0000 7fb59cff9640 10 client.4385.objecter ms_handle_connect 0x7fb58c0725d0 2024-03-29T10:43:56.128 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.140+0000 7fb5a684e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:56.128 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.140+0000 7fb5a684e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:56.128 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.140+0000 7fb5a684e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:43:56.128 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.140+0000 7fb5a684e640 1 -- 172.21.15.57:0/732001652 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a01351b0 con 0x7fb5a00b0860 2024-03-29T10:43:56.131 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.148+0000 7fb59cff9640 1 -- 172.21.15.57:0/732001652 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb59404ef60 con 0x7fb5a00b0860 2024-03-29T10:43:56.131 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.148+0000 7fb59cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:56.131 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.148+0000 7fb59cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:56.465 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.482+0000 7fb5a684e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:56.465 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.482+0000 7fb5a684e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:56.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.482+0000 7fb5a684e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:43:56.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.482+0000 7fb5a684e640 1 -- 172.21.15.57:0/732001652 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5a0075f10 con 0x7fb5a00b0860 2024-03-29T10:43:56.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.483+0000 7fb59cff9640 1 -- 172.21.15.57:0/732001652 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7fb594050e90 con 0x7fb5a00b0860 2024-03-29T10:43:56.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.483+0000 7fb59cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:56.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.483+0000 7fb59cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:56.467 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:43:56.467 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:56.471 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.487+0000 7fb5a684e640 10 librados: watch_flush enter 2024-03-29T10:43:56.471 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.487+0000 7fb5a684e640 10 librados: watch_flush exit 2024-03-29T10:43:56.471 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.487+0000 7fb5a684e640 20 client.4385.objecter shutdown clearing up homeless session... 2024-03-29T10:43:56.471 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.487+0000 7fb5a684e640 10 client.4385.objecter successfully canceled tick 2024-03-29T10:43:56.471 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.487+0000 7fb5a684e640 1 -- 172.21.15.57:0/732001652 >> v1:172.21.15.179:6816/2698295936 conn(0x7fb58c0725d0 legacy=0x7fb58c074a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:56.471 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.487+0000 7fb5a684e640 10 monclient: shutdown 2024-03-29T10:43:56.471 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.488+0000 7fb5a684e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:56.471 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.488+0000 7fb5a684e640 1 -- 172.21.15.57:0/732001652 >> v1:172.21.15.57:6790/0 conn(0x7fb5a00b0860 legacy=0x7fb5a006f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:56.471 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.488+0000 7fb5a684e640 1 -- 172.21.15.57:0/732001652 shutdown_connections 2024-03-29T10:43:56.471 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.488+0000 7fb5a684e640 1 -- 172.21.15.57:0/732001652 >> 172.21.15.57:0/732001652 conn(0x7fb5a006b660 msgr2=0x7fb5a013e440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:56.472 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.488+0000 7fb5a684e640 1 -- 172.21.15.57:0/732001652 shutdown_connections 2024-03-29T10:43:56.472 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.488+0000 7fb5a684e640 1 -- 172.21.15.57:0/732001652 wait complete. 2024-03-29T10:43:56.472 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:43:56.488+0000 7fb5a684e640 1 librados: shutdown 2024-03-29T10:43:56.489 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:03.296 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:03.457 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.474+0000 7f494e43e640 10 monclient: get_monmap_and_config 2024-03-29T10:44:03.457 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.474+0000 7f494e43e640 10 monclient: build_initial_monmap 2024-03-29T10:44:03.457 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.474+0000 7f494e43e640 10 monclient: monmap: 2024-03-29T10:44:03.457 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:44:03.457 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:03.457 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:44:03.475137+0000 2024-03-29T10:44:03.457 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:44:03.475137+0000 2024-03-29T10:44:03.457 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:03.457 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:03.458 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:44:03.458 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:44:03.458 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:44:03.458 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:03.458 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.475+0000 7f494e43e640 1 Processor -- start 2024-03-29T10:44:03.458 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.476+0000 7f494e43e640 1 -- start start 2024-03-29T10:44:03.458 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.476+0000 7f494e43e640 10 monclient: init 2024-03-29T10:44:03.460 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.477+0000 7f494e43e640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:03.460 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.477+0000 7f494e43e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:44:03.460 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.477+0000 7f494e43e640 10 monclient(hunting): picked mon.noname-c con 0x7f49480b4790 addr v1:172.21.15.179:6789/0 2024-03-29T10:44:03.460 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.477+0000 7f494e43e640 10 monclient(hunting): picked mon.noname-a con 0x7f49480aea90 addr v1:172.21.15.57:6789/0 2024-03-29T10:44:03.460 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.477+0000 7f494e43e640 10 monclient(hunting): picked mon.noname-b con 0x7f49480b39f0 addr v1:172.21.15.57:6790/0 2024-03-29T10:44:03.460 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.477+0000 7f494e43e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49480b0c60 con 0x7f49480aea90 2024-03-29T10:44:03.460 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.477+0000 7f494e43e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49480af020 con 0x7f49480b4790 2024-03-29T10:44:03.460 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.477+0000 7f494e43e640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f494813fbd0 con 0x7f49480b39f0 2024-03-29T10:44:03.460 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.477+0000 7f494e43e640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:03.460 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.477+0000 7f494e43e640 10 monclient(hunting): authenticate will time out at 1195.653485s 2024-03-29T10:44:03.461 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.477+0000 7f49477fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f49480aea90 0x7f49480af540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:54840/0 (socket says 172.21.15.57:54840) 2024-03-29T10:44:03.461 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.477+0000 7f49477fe640 1 -- 172.21.15.57:0/990900428 learned_addr learned my addr 172.21.15.57:0/990900428 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:44:03.461 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.478+0000 7f4946ffd640 1 -- 172.21.15.57:0/990900428 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2742131235 0 0) 0x7f494813fbd0 con 0x7f49480b39f0 2024-03-29T10:44:03.461 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.478+0000 7f4946ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:03.461 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.478+0000 7f4946ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:03.461 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.478+0000 7f4946ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:03.461 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.478+0000 7f4946ffd640 10 monclient(hunting): my global_id is 4388 2024-03-29T10:44:03.461 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.478+0000 7f4946ffd640 1 -- 172.21.15.57:0/990900428 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4930003520 con 0x7f49480b39f0 2024-03-29T10:44:03.462 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.479+0000 7f4946ffd640 1 -- 172.21.15.57:0/990900428 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 756241087 0 0) 0x7f49480b0c60 con 0x7f49480aea90 2024-03-29T10:44:03.462 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.479+0000 7f4946ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:03.462 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.479+0000 7f4946ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:03.462 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.479+0000 7f4946ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:03.462 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.479+0000 7f4946ffd640 10 monclient(hunting): my global_id is 4371 2024-03-29T10:44:03.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.479+0000 7f4946ffd640 1 -- 172.21.15.57:0/990900428 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f494813fbd0 con 0x7f49480aea90 2024-03-29T10:44:03.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.479+0000 7f4946ffd640 1 -- 172.21.15.57:0/990900428 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 514179294 0 0) 0x7f4930003520 con 0x7f49480b39f0 2024-03-29T10:44:03.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.479+0000 7f4946ffd640 1 -- 172.21.15.57:0/990900428 >> v1:172.21.15.179:6789/0 conn(0x7f49480b4790 legacy=0x7f49480b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:03.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.479+0000 7f4946ffd640 1 -- 172.21.15.57:0/990900428 >> v1:172.21.15.57:6789/0 conn(0x7f49480aea90 legacy=0x7f49480af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:03.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.479+0000 7f4946ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:44:03.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.479+0000 7f4946ffd640 1 monclient: found mon.noname-b 2024-03-29T10:44:03.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.479+0000 7f4946ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:03.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.479+0000 7f4946ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:44:03.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.479+0000 7f4946ffd640 1 -- 172.21.15.57:0/990900428 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494813fd60 con 0x7f49480b39f0 2024-03-29T10:44:03.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.479+0000 7f4946ffd640 10 monclient: _finish_auth 0 2024-03-29T10:44:03.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.479+0000 7f4946ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:44:03.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.479+0000 7f4946ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:03.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.479+0000 7f4946ffd640 1 -- 172.21.15.57:0/990900428 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f4938001610 con 0x7f49480b39f0 2024-03-29T10:44:03.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.479+0000 7f4946ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:03.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.479+0000 7f4946ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:03.464 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.479+0000 7f4946ffd640 10 monclient: dump: 2024-03-29T10:44:03.464 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:03.464 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:03.464 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:03.464 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:03.464 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:03.464 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:03.464 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:03.464 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:03.464 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:03.464 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:03.464 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.479+0000 7f494e43e640 5 monclient: authenticate success, global_id 4388 2024-03-29T10:44:03.464 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.479+0000 7f494e43e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:03.465 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.480+0000 7f4946ffd640 1 -- 172.21.15.57:0/990900428 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4938001830 con 0x7f49480b39f0 2024-03-29T10:44:03.465 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.480+0000 7f4946ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:03.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.480+0000 7f4946ffd640 1 -- 172.21.15.57:0/990900428 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f4938001bc0 con 0x7f49480b39f0 2024-03-29T10:44:03.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.480+0000 7f4946ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:03.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.480+0000 7f4946ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:03.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.480+0000 7f4946ffd640 10 monclient: dump: 2024-03-29T10:44:03.466 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:03.466 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:03.466 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:03.466 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:03.466 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:03.466 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:03.466 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:03.466 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:03.466 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:03.466 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:03.467 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.480+0000 7f494e43e640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:03.467 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.480+0000 7f494e43e640 4 set_mon_vals no callback set 2024-03-29T10:44:03.467 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.481+0000 7f494e43e640 10 monclient: shutdown 2024-03-29T10:44:03.467 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.481+0000 7f494e43e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:03.467 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.481+0000 7f494e43e640 1 -- 172.21.15.57:0/990900428 >> v1:172.21.15.57:6790/0 conn(0x7f49480b39f0 legacy=0x7f494813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:03.467 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.481+0000 7f494e43e640 1 -- 172.21.15.57:0/990900428 shutdown_connections 2024-03-29T10:44:03.467 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.481+0000 7f494e43e640 1 -- 172.21.15.57:0/990900428 >> 172.21.15.57:0/990900428 conn(0x7f4948066a00 msgr2=0x7f4948066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:03.467 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.481+0000 7f494e43e640 1 -- 172.21.15.57:0/990900428 shutdown_connections 2024-03-29T10:44:03.467 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.481+0000 7f494e43e640 1 -- 172.21.15.57:0/990900428 wait complete. 2024-03-29T10:44:03.467 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.482+0000 7f494e43e640 10 monclient: build_initial_monmap 2024-03-29T10:44:03.467 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.482+0000 7f494e43e640 10 monclient: monmap: 2024-03-29T10:44:03.467 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:44:03.467 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:03.467 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:44:03.467 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:44:03.468 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:03.468 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:03.468 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:44:03.468 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:44:03.468 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:44:03.468 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:03.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.483+0000 7f494e43e640 1 librados: starting msgr at 2024-03-29T10:44:03.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.483+0000 7f494e43e640 1 librados: starting objecter 2024-03-29T10:44:03.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.483+0000 7f494e43e640 1 Processor -- start 2024-03-29T10:44:03.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.483+0000 7f494e43e640 1 -- start start 2024-03-29T10:44:03.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.483+0000 7f494e43e640 1 librados: setting wanted keys 2024-03-29T10:44:03.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.483+0000 7f494e43e640 1 librados: calling monclient init 2024-03-29T10:44:03.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.483+0000 7f494e43e640 10 monclient: init 2024-03-29T10:44:03.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.484+0000 7f494e43e640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:03.473 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.484+0000 7f494e43e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:44:03.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.485+0000 7f494e43e640 10 monclient(hunting): picked mon.noname-a con 0x7f49480b39f0 addr v1:172.21.15.57:6789/0 2024-03-29T10:44:03.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.485+0000 7f494e43e640 10 monclient(hunting): picked mon.noname-c con 0x7f49480b4790 addr v1:172.21.15.57:6790/0 2024-03-29T10:44:03.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.485+0000 7f494e43e640 10 monclient(hunting): picked mon.noname-b con 0x7f4948141d10 addr v1:172.21.15.179:6789/0 2024-03-29T10:44:03.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.485+0000 7f494c9db640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f4948141d10 0x7f49481e1e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:48226/0 (socket says 172.21.15.57:48226) 2024-03-29T10:44:03.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.485+0000 7f494c9db640 1 -- 172.21.15.57:0/3400887861 learned_addr learned my addr 172.21.15.57:0/3400887861 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:44:03.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.485+0000 7f494e43e640 1 -- 172.21.15.57:0/3400887861 --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49480aae90 con 0x7f49480b39f0 2024-03-29T10:44:03.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.485+0000 7f4944ff9640 10 client.?.objecter ms_handle_connect 0x7f4948141d10 2024-03-29T10:44:03.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.485+0000 7f4944ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:03.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.485+0000 7f494e43e640 1 -- 172.21.15.57:0/3400887861 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49480ae5a0 con 0x7f4948141d10 2024-03-29T10:44:03.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.485+0000 7f494e43e640 1 -- 172.21.15.57:0/3400887861 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49480ad6f0 con 0x7f49480b4790 2024-03-29T10:44:03.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.485+0000 7f494e43e640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:03.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.486+0000 7f494e43e640 10 monclient(hunting): authenticate will time out at 1195.662083s 2024-03-29T10:44:03.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.486+0000 7f4944ff9640 10 client.?.objecter ms_handle_connect 0x7f49480b4790 2024-03-29T10:44:03.474 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.486+0000 7f4944ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:03.475 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.486+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606264976 0 0) 0x7f49480ae5a0 con 0x7f4948141d10 2024-03-29T10:44:03.475 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.486+0000 7f4944ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:03.475 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.486+0000 7f4944ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:03.475 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.486+0000 7f4944ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:03.475 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.486+0000 7f4944ff9640 10 monclient(hunting): my global_id is 4381 2024-03-29T10:44:03.475 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.486+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4934004eb0 con 0x7f4948141d10 2024-03-29T10:44:03.475 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.486+0000 7f4944ff9640 10 client.?.objecter ms_handle_connect 0x7f49480b39f0 2024-03-29T10:44:03.475 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.486+0000 7f4944ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:03.475 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.486+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1390898695 0 0) 0x7f49480ad6f0 con 0x7f49480b4790 2024-03-29T10:44:03.475 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.486+0000 7f4944ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:03.475 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.486+0000 7f4944ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:03.475 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.486+0000 7f4944ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:03.475 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.486+0000 7f4944ff9640 10 monclient(hunting): my global_id is 4391 2024-03-29T10:44:03.475 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.486+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49480ae5a0 con 0x7f49480b4790 2024-03-29T10:44:03.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.487+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862331202 0 0) 0x7f49480aae90 con 0x7f49480b39f0 2024-03-29T10:44:03.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.487+0000 7f4944ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:03.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.487+0000 7f4944ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:03.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.487+0000 7f4944ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:03.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.487+0000 7f4944ff9640 10 monclient(hunting): my global_id is 4374 2024-03-29T10:44:03.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.487+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49480ad6f0 con 0x7f49480b39f0 2024-03-29T10:44:03.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.487+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 922685376 0 0) 0x7f4934004eb0 con 0x7f4948141d10 2024-03-29T10:44:03.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49480aae90 con 0x7f4948141d10 2024-03-29T10:44:03.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f4938001610 con 0x7f4948141d10 2024-03-29T10:44:03.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:03.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:03.476 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 10 monclient(hunting): dump: 2024-03-29T10:44:03.476 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:03.476 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:03.476 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:03.477 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:03.477 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:03.477 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:03.477 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:03.477 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:03.477 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:03.477 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:03.477 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:03.477 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3288786561 0 0) 0x7f49480ad6f0 con 0x7f49480b39f0 2024-03-29T10:44:03.477 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4934004eb0 con 0x7f49480b39f0 2024-03-29T10:44:03.477 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f4940001a40 con 0x7f49480b39f0 2024-03-29T10:44:03.477 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:03.477 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:03.477 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 10 monclient(hunting): dump: 2024-03-29T10:44:03.477 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:03.478 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:03.478 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:03.478 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:03.478 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:03.478 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:03.478 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:03.478 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:03.478 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:03.478 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:03.478 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3283449200 0 0) 0x7f49480ae5a0 con 0x7f49480b4790 2024-03-29T10:44:03.478 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49480ad6f0 con 0x7f49480b4790 2024-03-29T10:44:03.478 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f493c001930 con 0x7f49480b4790 2024-03-29T10:44:03.478 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:03.478 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:03.479 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 10 monclient(hunting): dump: 2024-03-29T10:44:03.479 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:03.479 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:03.479 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:03.479 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:03.479 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:03.479 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:03.479 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:03.479 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:03.479 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:03.479 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:03.479 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 919245563 0 0) 0x7f49480aae90 con 0x7f4948141d10 2024-03-29T10:44:03.479 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 >> v1:172.21.15.57:6790/0 conn(0x7f49480b4790 legacy=0x7f49480acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:03.479 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 >> v1:172.21.15.57:6789/0 conn(0x7f49480b39f0 legacy=0x7f49480ab320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:03.480 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:44:03.480 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 1 monclient: found mon.b 2024-03-29T10:44:03.480 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:03.480 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:44:03.480 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49480aba60 con 0x7f4948141d10 2024-03-29T10:44:03.480 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 10 monclient: _finish_auth 0 2024-03-29T10:44:03.480 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:44:03.480 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f4944ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:03.480 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f494e43e640 5 monclient: authenticate success, global_id 4381 2024-03-29T10:44:03.480 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f494e43e640 10 monclient: _renew_subs 2024-03-29T10:44:03.480 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f494e43e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:44:03.480 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.488+0000 7f494e43e640 1 -- 172.21.15.57:0/3400887861 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49480ac940 con 0x7f4948141d10 2024-03-29T10:44:03.480 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.489+0000 7f494e43e640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:03.480 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.489+0000 7f494e43e640 10 monclient: _renew_subs 2024-03-29T10:44:03.480 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.489+0000 7f494e43e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:44:03.480 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.489+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49380038a0 con 0x7f4948141d10 2024-03-29T10:44:03.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.489+0000 7f494e43e640 1 -- 172.21.15.57:0/3400887861 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49480aa4f0 con 0x7f4948141d10 2024-03-29T10:44:03.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.489+0000 7f494e43e640 1 librados: init done 2024-03-29T10:44:03.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.489+0000 7f4944ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:03.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.489+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f4938003cb0 con 0x7f4948141d10 2024-03-29T10:44:03.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.489+0000 7f4944ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:03.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.489+0000 7f4944ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:03.481 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.489+0000 7f4944ff9640 10 monclient: dump: 2024-03-29T10:44:03.481 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:03.481 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:03.481 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:03.481 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:03.481 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:03.481 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:03.481 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:03.481 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:03.482 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:03.482 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:03.482 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.489+0000 7f49467fc640 4 set_mon_vals no callback set 2024-03-29T10:44:03.483 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.491+0000 7f494e43e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:03.483 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.491+0000 7f494e43e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:03.483 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.491+0000 7f494e43e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:44:03.483 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.491+0000 7f494e43e640 1 -- 172.21.15.57:0/3400887861 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4948075f10 con 0x7f4948141d10 2024-03-29T10:44:03.483 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.492+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f4938003e10 con 0x7f4948141d10 2024-03-29T10:44:03.483 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.493+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f4938084ae0 con 0x7f4948141d10 2024-03-29T10:44:03.484 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.493+0000 7f4944ff9640 10 client.4381.objecter ms_dispatch 0x7f4948000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:03.484 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.493+0000 7f4944ff9640 3 client.4381.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:03.484 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.493+0000 7f4944ff9640 3 client.4381.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:03.484 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.493+0000 7f4944ff9640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-29T10:44:03.484 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.493+0000 7f4947fff640 10 monclient: get_auth_request con 0x7f493406e1d0 auth_method 0 2024-03-29T10:44:03.484 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.493+0000 7f4944ff9640 10 client.4381.objecter ms_handle_connect 0x7f493406e1d0 2024-03-29T10:44:03.484 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.498+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f49380503c0 con 0x7f4948141d10 2024-03-29T10:44:03.484 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.498+0000 7f4944ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:03.484 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.498+0000 7f4944ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:03.871 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.888+0000 7f494e43e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:03.871 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.888+0000 7f494e43e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:03.872 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.889+0000 7f494e43e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:44:03.872 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.889+0000 7f494e43e640 1 -- 172.21.15.57:0/3400887861 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f494813e760 con 0x7f4948141d10 2024-03-29T10:44:03.872 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.890+0000 7f4944ff9640 1 -- 172.21.15.57:0/3400887861 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f49380522f0 con 0x7f4948141d10 2024-03-29T10:44:03.872 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.890+0000 7f4944ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:03.873 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.890+0000 7f4944ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:03.873 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:44:03.873 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:03.876 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.893+0000 7f494e43e640 10 librados: watch_flush enter 2024-03-29T10:44:03.876 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.893+0000 7f494e43e640 10 librados: watch_flush exit 2024-03-29T10:44:03.876 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.893+0000 7f494e43e640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-29T10:44:03.876 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.893+0000 7f494e43e640 10 client.4381.objecter successfully canceled tick 2024-03-29T10:44:03.876 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.893+0000 7f494e43e640 1 -- 172.21.15.57:0/3400887861 >> v1:172.21.15.179:6816/2698295936 conn(0x7f493406e1d0 legacy=0x7f4934070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:03.876 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.893+0000 7f494e43e640 10 monclient: shutdown 2024-03-29T10:44:03.877 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.893+0000 7f494e43e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:03.877 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.893+0000 7f494e43e640 1 -- 172.21.15.57:0/3400887861 >> v1:172.21.15.179:6789/0 conn(0x7f4948141d10 legacy=0x7f49481e1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:03.877 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.893+0000 7f494e43e640 1 -- 172.21.15.57:0/3400887861 shutdown_connections 2024-03-29T10:44:03.877 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.893+0000 7f494e43e640 1 -- 172.21.15.57:0/3400887861 >> 172.21.15.57:0/3400887861 conn(0x7f4948066a00 msgr2=0x7f494813f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:03.877 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.894+0000 7f494e43e640 1 -- 172.21.15.57:0/3400887861 shutdown_connections 2024-03-29T10:44:03.877 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.894+0000 7f494e43e640 1 -- 172.21.15.57:0/3400887861 wait complete. 2024-03-29T10:44:03.877 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:03.894+0000 7f494e43e640 1 librados: shutdown 2024-03-29T10:44:03.891 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:10.697 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:10.832 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.849+0000 7f06379f4640 10 monclient: get_monmap_and_config 2024-03-29T10:44:10.832 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.849+0000 7f06379f4640 10 monclient: build_initial_monmap 2024-03-29T10:44:10.832 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.849+0000 7f06379f4640 10 monclient: monmap: 2024-03-29T10:44:10.832 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:44:10.833 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:10.833 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:44:10.850415+0000 2024-03-29T10:44:10.833 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:44:10.850415+0000 2024-03-29T10:44:10.833 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:10.833 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:10.833 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:44:10.833 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:44:10.833 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:44:10.833 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:10.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.850+0000 7f06379f4640 1 Processor -- start 2024-03-29T10:44:10.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.850+0000 7f06379f4640 1 -- start start 2024-03-29T10:44:10.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.850+0000 7f06379f4640 10 monclient: init 2024-03-29T10:44:10.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.851+0000 7f06379f4640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:10.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.851+0000 7f06379f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:10.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.851+0000 7f06379f4640 10 monclient(hunting): picked mon.noname-b con 0x7f06300a9750 addr v1:172.21.15.57:6790/0 2024-03-29T10:44:10.836 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.851+0000 7f0635790640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f06300a9750 0x7f063013dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:35978/0 (socket says 172.21.15.57:35978) 2024-03-29T10:44:10.836 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.851+0000 7f06379f4640 10 monclient(hunting): picked mon.noname-a con 0x7f063013e1e0 addr v1:172.21.15.57:6789/0 2024-03-29T10:44:10.836 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.852+0000 7f06379f4640 10 monclient(hunting): picked mon.noname-c con 0x7f0630071c30 addr v1:172.21.15.179:6789/0 2024-03-29T10:44:10.836 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.852+0000 7f06379f4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06300ab300 con 0x7f063013e1e0 2024-03-29T10:44:10.836 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.852+0000 7f0635790640 1 -- 172.21.15.57:0/3081477216 learned_addr learned my addr 172.21.15.57:0/3081477216 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:44:10.836 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.852+0000 7f06379f4640 1 -- 172.21.15.57:0/3081477216 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06300ac2f0 con 0x7f0630071c30 2024-03-29T10:44:10.836 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.852+0000 7f06379f4640 1 -- 172.21.15.57:0/3081477216 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06300ad8d0 con 0x7f06300a9750 2024-03-29T10:44:10.836 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.852+0000 7f06379f4640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:10.836 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.852+0000 7f06379f4640 10 monclient(hunting): authenticate will time out at 1203.028247s 2024-03-29T10:44:10.836 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.852+0000 7f0627fff640 1 -- 172.21.15.57:0/3081477216 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532300139 0 0) 0x7f06300ad8d0 con 0x7f06300a9750 2024-03-29T10:44:10.836 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.852+0000 7f0627fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:10.836 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.852+0000 7f0627fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:10.836 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.852+0000 7f0627fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:10.836 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.852+0000 7f0627fff640 10 monclient(hunting): my global_id is 4394 2024-03-29T10:44:10.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.852+0000 7f0627fff640 1 -- 172.21.15.57:0/3081477216 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f061c003520 con 0x7f06300a9750 2024-03-29T10:44:10.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.853+0000 7f0627fff640 1 -- 172.21.15.57:0/3081477216 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2117426678 0 0) 0x7f061c003520 con 0x7f06300a9750 2024-03-29T10:44:10.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.853+0000 7f0627fff640 1 -- 172.21.15.57:0/3081477216 >> v1:172.21.15.179:6789/0 conn(0x7f0630071c30 legacy=0x7f06300ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:10.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.853+0000 7f0627fff640 1 -- 172.21.15.57:0/3081477216 >> v1:172.21.15.57:6789/0 conn(0x7f063013e1e0 legacy=0x7f06300ac710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:10.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.855+0000 7f0627fff640 10 monclient: _finish_hunting 0 2024-03-29T10:44:10.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.855+0000 7f0627fff640 1 monclient: found mon.noname-b 2024-03-29T10:44:10.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.855+0000 7f0627fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:10.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.855+0000 7f0627fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:44:10.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.855+0000 7f0627fff640 1 -- 172.21.15.57:0/3081477216 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06300ada60 con 0x7f06300a9750 2024-03-29T10:44:10.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.855+0000 7f0627fff640 10 monclient: _finish_auth 0 2024-03-29T10:44:10.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.855+0000 7f0627fff640 10 monclient: _check_auth_tickets 2024-03-29T10:44:10.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.855+0000 7f0627fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:10.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.855+0000 7f0627fff640 1 -- 172.21.15.57:0/3081477216 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f062c001690 con 0x7f06300a9750 2024-03-29T10:44:10.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.855+0000 7f0627fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:10.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.855+0000 7f0627fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:10.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.855+0000 7f0627fff640 10 monclient: dump: 2024-03-29T10:44:10.838 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:10.838 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:10.839 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:10.839 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:10.839 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:10.839 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:10.839 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:10.839 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:10.839 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:10.839 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:10.839 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.855+0000 7f06379f4640 5 monclient: authenticate success, global_id 4394 2024-03-29T10:44:10.839 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.855+0000 7f06379f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:10.840 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.855+0000 7f0627fff640 1 -- 172.21.15.57:0/3081477216 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f062c001920 con 0x7f06300a9750 2024-03-29T10:44:10.840 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.855+0000 7f0627fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:10.841 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.855+0000 7f0627fff640 1 -- 172.21.15.57:0/3081477216 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f062c001d30 con 0x7f06300a9750 2024-03-29T10:44:10.841 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.855+0000 7f0627fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:10.841 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.855+0000 7f0627fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:10.841 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.855+0000 7f0627fff640 10 monclient: dump: 2024-03-29T10:44:10.841 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:10.841 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:10.841 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:10.841 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:10.841 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:10.841 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:10.841 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:10.841 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:10.841 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:10.841 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:10.841 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.855+0000 7f06379f4640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:10.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.855+0000 7f06379f4640 4 set_mon_vals no callback set 2024-03-29T10:44:10.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.856+0000 7f06379f4640 10 monclient: shutdown 2024-03-29T10:44:10.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.856+0000 7f06379f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:10.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.856+0000 7f06379f4640 1 -- 172.21.15.57:0/3081477216 >> v1:172.21.15.57:6790/0 conn(0x7f06300a9750 legacy=0x7f063013dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:10.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.857+0000 7f06379f4640 1 -- 172.21.15.57:0/3081477216 shutdown_connections 2024-03-29T10:44:10.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.857+0000 7f06379f4640 1 -- 172.21.15.57:0/3081477216 >> 172.21.15.57:0/3081477216 conn(0x7f063006b660 msgr2=0x7f063006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:10.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.857+0000 7f06379f4640 1 -- 172.21.15.57:0/3081477216 shutdown_connections 2024-03-29T10:44:10.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.857+0000 7f06379f4640 1 -- 172.21.15.57:0/3081477216 wait complete. 2024-03-29T10:44:10.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.858+0000 7f06379f4640 10 monclient: build_initial_monmap 2024-03-29T10:44:10.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.858+0000 7f06379f4640 10 monclient: monmap: 2024-03-29T10:44:10.842 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:44:10.842 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:10.842 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:44:10.842 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:44:10.843 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:10.843 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:10.843 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:44:10.843 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:44:10.843 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:44:10.843 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:10.845 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.858+0000 7f06379f4640 1 librados: starting msgr at 2024-03-29T10:44:10.845 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.858+0000 7f06379f4640 1 librados: starting objecter 2024-03-29T10:44:10.845 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.858+0000 7f06379f4640 1 Processor -- start 2024-03-29T10:44:10.845 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.858+0000 7f06379f4640 1 -- start start 2024-03-29T10:44:10.845 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.858+0000 7f06379f4640 1 librados: setting wanted keys 2024-03-29T10:44:10.845 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.858+0000 7f06379f4640 1 librados: calling monclient init 2024-03-29T10:44:10.845 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.858+0000 7f06379f4640 10 monclient: init 2024-03-29T10:44:10.845 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.859+0000 7f06379f4640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:10.845 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.859+0000 7f06379f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:44:10.845 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.859+0000 7f06379f4640 10 monclient(hunting): picked mon.noname-a con 0x7f0630071c30 addr v1:172.21.15.57:6789/0 2024-03-29T10:44:10.845 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.859+0000 7f06379f4640 10 monclient(hunting): picked mon.noname-c con 0x7f06300a9750 addr v1:172.21.15.57:6790/0 2024-03-29T10:44:10.845 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.859+0000 7f06379f4640 10 monclient(hunting): picked mon.noname-b con 0x7f063013e1e0 addr v1:172.21.15.179:6789/0 2024-03-29T10:44:10.845 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.859+0000 7f06379f4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0630122910 con 0x7f0630071c30 2024-03-29T10:44:10.846 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.859+0000 7f06379f4640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0630071740 con 0x7f063013e1e0 2024-03-29T10:44:10.846 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.859+0000 7f06379f4640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0630121a90 con 0x7f06300a9750 2024-03-29T10:44:10.846 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.859+0000 7f06379f4640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:10.846 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.859+0000 7f06379f4640 10 monclient(hunting): authenticate will time out at 1203.035485s 2024-03-29T10:44:10.846 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.859+0000 7f0634f8f640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f06300a9750 0x7f0630120ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:36128/0 (socket says 172.21.15.57:36128) 2024-03-29T10:44:10.846 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.859+0000 7f0634f8f640 1 -- 172.21.15.57:0/2932427648 learned_addr learned my addr 172.21.15.57:0/2932427648 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:44:10.846 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.859+0000 7f0625ffb640 10 client.?.objecter ms_handle_connect 0x7f06300a9750 2024-03-29T10:44:10.846 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.859+0000 7f0625ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:10.846 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.859+0000 7f0625ffb640 1 -- 172.21.15.57:0/2932427648 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836621807 0 0) 0x7f0630121a90 con 0x7f06300a9750 2024-03-29T10:44:10.846 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.859+0000 7f0625ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:10.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.859+0000 7f0625ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:10.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.859+0000 7f0625ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:10.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.859+0000 7f0625ffb640 10 monclient(hunting): my global_id is 4397 2024-03-29T10:44:10.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.860+0000 7f0625ffb640 1 -- 172.21.15.57:0/2932427648 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0614004eb0 con 0x7f06300a9750 2024-03-29T10:44:10.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.860+0000 7f0625ffb640 10 client.?.objecter ms_handle_connect 0x7f063013e1e0 2024-03-29T10:44:10.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.860+0000 7f0625ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:10.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.860+0000 7f0625ffb640 1 -- 172.21.15.57:0/2932427648 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3290869232 0 0) 0x7f0614004eb0 con 0x7f06300a9750 2024-03-29T10:44:10.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.860+0000 7f0625ffb640 1 -- 172.21.15.57:0/2932427648 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0630121a90 con 0x7f06300a9750 2024-03-29T10:44:10.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.860+0000 7f0625ffb640 1 -- 172.21.15.57:0/2932427648 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f06200017d0 con 0x7f06300a9750 2024-03-29T10:44:10.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.860+0000 7f0625ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:10.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.860+0000 7f0625ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:10.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.860+0000 7f0625ffb640 10 monclient(hunting): dump: 2024-03-29T10:44:10.847 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:10.847 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:10.847 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:10.848 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:10.848 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:10.848 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:10.848 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:10.848 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:10.848 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:10.848 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:10.848 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.861+0000 7f0625ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:10.848 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.861+0000 7f0625ffb640 1 -- 172.21.15.57:0/2932427648 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4044293930 0 0) 0x7f0630121a90 con 0x7f06300a9750 2024-03-29T10:44:10.848 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.861+0000 7f0625ffb640 1 -- 172.21.15.57:0/2932427648 >> v1:172.21.15.179:6789/0 conn(0x7f063013e1e0 legacy=0x7f06301e21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:10.848 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.861+0000 7f0625ffb640 1 -- 172.21.15.57:0/2932427648 >> v1:172.21.15.57:6789/0 conn(0x7f0630071c30 legacy=0x7f0630123380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:44:10.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.862+0000 7f0625ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:44:10.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.862+0000 7f0625ffb640 1 monclient: found mon.c 2024-03-29T10:44:10.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.862+0000 7f0625ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:10.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.862+0000 7f0625ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:44:10.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.862+0000 7f0625ffb640 1 -- 172.21.15.57:0/2932427648 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06301215f0 con 0x7f06300a9750 2024-03-29T10:44:10.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.862+0000 7f0625ffb640 10 monclient: _finish_auth 0 2024-03-29T10:44:10.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.862+0000 7f0625ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:44:10.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.862+0000 7f0625ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:10.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.862+0000 7f06379f4640 5 monclient: authenticate success, global_id 4397 2024-03-29T10:44:10.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.863+0000 7f06379f4640 10 monclient: _renew_subs 2024-03-29T10:44:10.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.863+0000 7f06379f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:44:10.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.863+0000 7f06379f4640 1 -- 172.21.15.57:0/2932427648 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06300b2580 con 0x7f06300a9750 2024-03-29T10:44:10.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.863+0000 7f0625ffb640 1 -- 172.21.15.57:0/2932427648 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0620001d60 con 0x7f06300a9750 2024-03-29T10:44:10.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.863+0000 7f0625ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:10.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.863+0000 7f0625ffb640 1 -- 172.21.15.57:0/2932427648 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f0620004a00 con 0x7f06300a9750 2024-03-29T10:44:10.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.863+0000 7f0625ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:10.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.863+0000 7f0625ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:10.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.863+0000 7f0625ffb640 10 monclient: dump: 2024-03-29T10:44:10.851 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:10.851 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:10.851 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:10.851 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:10.851 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:10.851 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:10.851 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:10.851 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:10.851 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:10.852 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:10.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.863+0000 7f06379f4640 10 client.4397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:10.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.863+0000 7f06277fe640 4 set_mon_vals no callback set 2024-03-29T10:44:10.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.863+0000 7f06379f4640 10 monclient: _renew_subs 2024-03-29T10:44:10.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.863+0000 7f06379f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:44:10.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.863+0000 7f06379f4640 1 -- 172.21.15.57:0/2932427648 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06301e3ab0 con 0x7f06300a9750 2024-03-29T10:44:10.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.863+0000 7f06379f4640 1 librados: init done 2024-03-29T10:44:10.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.865+0000 7f06379f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:10.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.865+0000 7f0625ffb640 1 -- 172.21.15.57:0/2932427648 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f062001a9d0 con 0x7f06300a9750 2024-03-29T10:44:10.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.865+0000 7f06379f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:10.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.865+0000 7f06379f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:44:10.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.865+0000 7f06379f4640 1 -- 172.21.15.57:0/2932427648 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06301205d0 con 0x7f06300a9750 2024-03-29T10:44:10.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.866+0000 7f0625ffb640 1 -- 172.21.15.57:0/2932427648 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f0620084a30 con 0x7f06300a9750 2024-03-29T10:44:10.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.866+0000 7f0625ffb640 10 client.4397.objecter ms_dispatch 0x7f0630000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:10.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.866+0000 7f0625ffb640 3 client.4397.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:10.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.866+0000 7f0625ffb640 3 client.4397.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:10.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.866+0000 7f0625ffb640 20 client.4397.objecter dump_active .. 0 homeless 2024-03-29T10:44:10.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.868+0000 7f0635790640 10 monclient: get_auth_request con 0x7f061406dc70 auth_method 0 2024-03-29T10:44:10.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.868+0000 7f0625ffb640 10 client.4397.objecter ms_handle_connect 0x7f061406dc70 2024-03-29T10:44:10.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.874+0000 7f0625ffb640 1 -- 172.21.15.57:0/2932427648 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0620050310 con 0x7f06300a9750 2024-03-29T10:44:10.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.874+0000 7f0625ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:10.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:10.874+0000 7f0625ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:11.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:11.197+0000 7f06379f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:11.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:11.197+0000 7f06379f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:11.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:11.197+0000 7f06379f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:44:11.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:11.197+0000 7f06379f4640 1 -- 172.21.15.57:0/2932427648 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0630075f10 con 0x7f06300a9750 2024-03-29T10:44:11.183 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:11.200+0000 7f0625ffb640 1 -- 172.21.15.57:0/2932427648 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f0620052240 con 0x7f06300a9750 2024-03-29T10:44:11.184 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:11.200+0000 7f0625ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:11.184 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:11.200+0000 7f0625ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:11.184 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:44:11.184 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:11.186 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:11.203+0000 7f06379f4640 10 librados: watch_flush enter 2024-03-29T10:44:11.186 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:11.203+0000 7f06379f4640 10 librados: watch_flush exit 2024-03-29T10:44:11.186 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:11.203+0000 7f06379f4640 20 client.4397.objecter shutdown clearing up homeless session... 2024-03-29T10:44:11.186 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:11.203+0000 7f06379f4640 10 client.4397.objecter successfully canceled tick 2024-03-29T10:44:11.187 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:11.203+0000 7f06379f4640 1 -- 172.21.15.57:0/2932427648 >> v1:172.21.15.179:6816/2698295936 conn(0x7f061406dc70 legacy=0x7f0614070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:11.187 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:11.203+0000 7f06379f4640 10 monclient: shutdown 2024-03-29T10:44:11.187 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:11.203+0000 7f06379f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:11.187 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:11.203+0000 7f06379f4640 1 -- 172.21.15.57:0/2932427648 >> v1:172.21.15.57:6790/0 conn(0x7f06300a9750 legacy=0x7f0630120ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:11.187 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:11.203+0000 7f06379f4640 1 -- 172.21.15.57:0/2932427648 shutdown_connections 2024-03-29T10:44:11.187 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:11.203+0000 7f06379f4640 1 -- 172.21.15.57:0/2932427648 >> 172.21.15.57:0/2932427648 conn(0x7f063006b660 msgr2=0x7f06300b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:11.187 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:11.203+0000 7f06379f4640 1 -- 172.21.15.57:0/2932427648 shutdown_connections 2024-03-29T10:44:11.187 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:11.203+0000 7f06379f4640 1 -- 172.21.15.57:0/2932427648 wait complete. 2024-03-29T10:44:11.187 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:11.204+0000 7f06379f4640 1 librados: shutdown 2024-03-29T10:44:11.201 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:18.008 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:18.156 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.172+0000 7feafc965640 10 monclient: get_monmap_and_config 2024-03-29T10:44:18.156 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.172+0000 7feafc965640 10 monclient: build_initial_monmap 2024-03-29T10:44:18.156 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.172+0000 7feafc965640 10 monclient: monmap: 2024-03-29T10:44:18.156 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:44:18.156 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:18.156 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:44:18.174126+0000 2024-03-29T10:44:18.156 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:44:18.174126+0000 2024-03-29T10:44:18.156 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:18.156 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:18.156 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:44:18.157 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:44:18.157 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:44:18.157 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:18.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.173+0000 7feafc965640 1 Processor -- start 2024-03-29T10:44:18.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.173+0000 7feafc965640 1 -- start start 2024-03-29T10:44:18.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.173+0000 7feafc965640 10 monclient: init 2024-03-29T10:44:18.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.174+0000 7feafc965640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:18.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.174+0000 7feafc965640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:44:18.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.174+0000 7feafc965640 10 monclient(hunting): picked mon.noname-c con 0x7feaf81422b0 addr v1:172.21.15.179:6789/0 2024-03-29T10:44:18.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.174+0000 7feafc965640 10 monclient(hunting): picked mon.noname-a con 0x7feaf8142690 addr v1:172.21.15.57:6789/0 2024-03-29T10:44:18.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.174+0000 7feafc965640 10 monclient(hunting): picked mon.noname-b con 0x7feaf80b17d0 addr v1:172.21.15.57:6790/0 2024-03-29T10:44:18.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.174+0000 7feafc965640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf8142c80 con 0x7feaf8142690 2024-03-29T10:44:18.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.174+0000 7feafc965640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf80aae40 con 0x7feaf81422b0 2024-03-29T10:44:18.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.174+0000 7feafc965640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf813fbd0 con 0x7feaf80b17d0 2024-03-29T10:44:18.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.174+0000 7feafc965640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:18.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.174+0000 7feafc965640 10 monclient(hunting): authenticate will time out at 1210.351535s 2024-03-29T10:44:18.161 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.177+0000 7feaf6d9d640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7feaf80b17d0 0x7feaf80b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:41748/0 (socket says 172.21.15.57:41748) 2024-03-29T10:44:18.162 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.177+0000 7feaf6d9d640 1 -- 172.21.15.57:0/1077185554 learned_addr learned my addr 172.21.15.57:0/1077185554 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:44:18.162 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.177+0000 7feaf559a640 1 -- 172.21.15.57:0/1077185554 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1415312393 0 0) 0x7feaf813fbd0 con 0x7feaf80b17d0 2024-03-29T10:44:18.162 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.177+0000 7feaf559a640 10 monclient(hunting): _negotiate 2024-03-29T10:44:18.162 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.177+0000 7feaf559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:18.162 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.177+0000 7feaf559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:18.162 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.177+0000 7feaf559a640 10 monclient(hunting): my global_id is 4400 2024-03-29T10:44:18.162 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.177+0000 7feaf559a640 1 -- 172.21.15.57:0/1077185554 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feae0003520 con 0x7feaf80b17d0 2024-03-29T10:44:18.162 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.177+0000 7feaf559a640 1 -- 172.21.15.57:0/1077185554 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2473191750 0 0) 0x7feae0003520 con 0x7feaf80b17d0 2024-03-29T10:44:18.162 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.177+0000 7feaf559a640 1 -- 172.21.15.57:0/1077185554 >> v1:172.21.15.179:6789/0 conn(0x7feaf81422b0 legacy=0x7feaf813f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:18.165 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.179+0000 7feaf559a640 1 -- 172.21.15.57:0/1077185554 >> v1:172.21.15.57:6789/0 conn(0x7feaf8142690 legacy=0x7feaf80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:18.165 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.179+0000 7feaf559a640 10 monclient: _finish_hunting 0 2024-03-29T10:44:18.165 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.179+0000 7feaf559a640 1 monclient: found mon.noname-b 2024-03-29T10:44:18.165 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.179+0000 7feaf559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:18.166 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.179+0000 7feaf559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:44:18.166 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.179+0000 7feaf559a640 1 -- 172.21.15.57:0/1077185554 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf8151940 con 0x7feaf80b17d0 2024-03-29T10:44:18.166 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.179+0000 7feaf559a640 10 monclient: _finish_auth 0 2024-03-29T10:44:18.166 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.179+0000 7feaf559a640 10 monclient: _check_auth_tickets 2024-03-29T10:44:18.166 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.179+0000 7feaf559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:18.166 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.179+0000 7feaf559a640 1 -- 172.21.15.57:0/1077185554 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7feae8001690 con 0x7feaf80b17d0 2024-03-29T10:44:18.166 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.179+0000 7feaf559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:18.166 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.179+0000 7feaf559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:18.166 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.179+0000 7feaf559a640 10 monclient: dump: 2024-03-29T10:44:18.166 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:18.166 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:18.166 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:18.166 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:18.166 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:18.167 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:18.167 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:18.167 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:18.167 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:18.167 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:18.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.179+0000 7feafc965640 5 monclient: authenticate success, global_id 4400 2024-03-29T10:44:18.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.179+0000 7feafc965640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:18.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.179+0000 7feaf559a640 1 -- 172.21.15.57:0/1077185554 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feae80017f0 con 0x7feaf80b17d0 2024-03-29T10:44:18.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.179+0000 7feaf559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:18.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.179+0000 7feaf559a640 1 -- 172.21.15.57:0/1077185554 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7feae8001c40 con 0x7feaf80b17d0 2024-03-29T10:44:18.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.179+0000 7feaf559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:18.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.179+0000 7feaf559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:18.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.179+0000 7feaf559a640 10 monclient: dump: 2024-03-29T10:44:18.167 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:18.167 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:18.168 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:18.168 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:18.168 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:18.168 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:18.168 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:18.168 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:18.168 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:18.168 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:18.168 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.179+0000 7feafc965640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:18.168 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.179+0000 7feafc965640 4 set_mon_vals no callback set 2024-03-29T10:44:18.168 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.180+0000 7feafc965640 10 monclient: shutdown 2024-03-29T10:44:18.168 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.180+0000 7feafc965640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:18.168 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.180+0000 7feafc965640 1 -- 172.21.15.57:0/1077185554 >> v1:172.21.15.57:6790/0 conn(0x7feaf80b17d0 legacy=0x7feaf80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:18.168 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.181+0000 7feafc965640 1 -- 172.21.15.57:0/1077185554 shutdown_connections 2024-03-29T10:44:18.169 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.181+0000 7feafc965640 1 -- 172.21.15.57:0/1077185554 >> 172.21.15.57:0/1077185554 conn(0x7feaf8066a00 msgr2=0x7feaf8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:18.169 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.181+0000 7feafc965640 1 -- 172.21.15.57:0/1077185554 shutdown_connections 2024-03-29T10:44:18.169 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.181+0000 7feafc965640 1 -- 172.21.15.57:0/1077185554 wait complete. 2024-03-29T10:44:18.169 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.182+0000 7feafc965640 10 monclient: build_initial_monmap 2024-03-29T10:44:18.169 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.182+0000 7feafc965640 10 monclient: monmap: 2024-03-29T10:44:18.169 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:44:18.169 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:18.170 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:44:18.170 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:44:18.170 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:18.170 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:18.170 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:44:18.170 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:44:18.170 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:44:18.170 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:18.170 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.182+0000 7feafc965640 1 librados: starting msgr at 2024-03-29T10:44:18.170 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.182+0000 7feafc965640 1 librados: starting objecter 2024-03-29T10:44:18.170 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.183+0000 7feafc965640 1 Processor -- start 2024-03-29T10:44:18.170 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.183+0000 7feafc965640 1 -- start start 2024-03-29T10:44:18.170 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.183+0000 7feafc965640 1 librados: setting wanted keys 2024-03-29T10:44:18.170 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.183+0000 7feafc965640 1 librados: calling monclient init 2024-03-29T10:44:18.171 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.183+0000 7feafc965640 10 monclient: init 2024-03-29T10:44:18.171 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.183+0000 7feafc965640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:18.171 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.183+0000 7feafc965640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:18.171 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.184+0000 7feafc965640 10 monclient(hunting): picked mon.noname-a con 0x7feaf80b17d0 addr v1:172.21.15.57:6789/0 2024-03-29T10:44:18.171 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.184+0000 7feafc965640 10 monclient(hunting): picked mon.noname-b con 0x7feaf81422b0 addr v1:172.21.15.179:6789/0 2024-03-29T10:44:18.171 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.184+0000 7feafc965640 10 monclient(hunting): picked mon.noname-c con 0x7feaf8142690 addr v1:172.21.15.57:6790/0 2024-03-29T10:44:18.171 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.184+0000 7feafc965640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf8132b30 con 0x7feaf80b17d0 2024-03-29T10:44:18.171 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.184+0000 7feafc965640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf80acc80 con 0x7feaf81422b0 2024-03-29T10:44:18.171 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.184+0000 7feafc965640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf80acf60 con 0x7feaf8142690 2024-03-29T10:44:18.171 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.184+0000 7feafc965640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:18.171 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.184+0000 7feafc965640 10 monclient(hunting): authenticate will time out at 1210.361849s 2024-03-29T10:44:18.173 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.187+0000 7feaf5d9b640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7feaf81422b0 0x7feaf8130580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:35727/0 (socket says 172.21.15.57:35727) 2024-03-29T10:44:18.173 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.187+0000 7feaf5d9b640 1 -- 172.21.15.57:0/2325283958 learned_addr learned my addr 172.21.15.57:0/2325283958 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:44:18.173 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.187+0000 7feae6ffd640 10 client.?.objecter ms_handle_connect 0x7feaf81422b0 2024-03-29T10:44:18.173 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.187+0000 7feae6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:18.173 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.187+0000 7feae6ffd640 1 -- 172.21.15.57:0/2325283958 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1621087953 0 0) 0x7feaf80acc80 con 0x7feaf81422b0 2024-03-29T10:44:18.173 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.187+0000 7feae6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:18.173 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.187+0000 7feae6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:18.173 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.187+0000 7feae6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:18.173 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.187+0000 7feae6ffd640 10 monclient(hunting): my global_id is 4387 2024-03-29T10:44:18.173 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.187+0000 7feae6ffd640 1 -- 172.21.15.57:0/2325283958 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fead8004eb0 con 0x7feaf81422b0 2024-03-29T10:44:18.174 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.187+0000 7feae6ffd640 10 client.?.objecter ms_handle_connect 0x7feaf8142690 2024-03-29T10:44:18.174 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.187+0000 7feae6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:18.174 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.188+0000 7feae6ffd640 1 -- 172.21.15.57:0/2325283958 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437188650 0 0) 0x7feaf80acf60 con 0x7feaf8142690 2024-03-29T10:44:18.174 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.188+0000 7feae6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:18.174 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.188+0000 7feae6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:18.174 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.188+0000 7feae6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:18.174 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.188+0000 7feae6ffd640 10 monclient(hunting): my global_id is 4403 2024-03-29T10:44:18.174 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.188+0000 7feae6ffd640 1 -- 172.21.15.57:0/2325283958 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaf80acc80 con 0x7feaf8142690 2024-03-29T10:44:18.174 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.188+0000 7feae6ffd640 1 -- 172.21.15.57:0/2325283958 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 154721860 0 0) 0x7feaf80acc80 con 0x7feaf8142690 2024-03-29T10:44:18.174 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.188+0000 7feae6ffd640 1 -- 172.21.15.57:0/2325283958 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feaf80acf60 con 0x7feaf8142690 2024-03-29T10:44:18.174 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.188+0000 7feae6ffd640 10 client.?.objecter ms_handle_connect 0x7feaf80b17d0 2024-03-29T10:44:18.174 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.188+0000 7feae6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:18.174 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.188+0000 7feae6ffd640 1 -- 172.21.15.57:0/2325283958 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7feae8001b00 con 0x7feaf8142690 2024-03-29T10:44:18.175 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.188+0000 7feae6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:18.175 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.188+0000 7feae6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:18.175 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.188+0000 7feae6ffd640 10 monclient(hunting): dump: 2024-03-29T10:44:18.175 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:18.175 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:18.175 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:18.175 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:18.175 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:18.175 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:18.175 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:18.175 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:18.175 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:18.175 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:18.175 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.188+0000 7feae6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:18.175 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.188+0000 7feae6ffd640 1 -- 172.21.15.57:0/2325283958 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225886656 0 0) 0x7feaf8132b30 con 0x7feaf80b17d0 2024-03-29T10:44:18.176 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.188+0000 7feae6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:18.176 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.188+0000 7feae6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:18.176 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.188+0000 7feae6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:18.176 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.188+0000 7feae6ffd640 10 monclient(hunting): my global_id is 4380 2024-03-29T10:44:18.176 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.188+0000 7feae6ffd640 1 -- 172.21.15.57:0/2325283958 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaf80acc80 con 0x7feaf80b17d0 2024-03-29T10:44:18.176 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feae6ffd640 1 -- 172.21.15.57:0/2325283958 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2209096919 0 0) 0x7feaf80acc80 con 0x7feaf80b17d0 2024-03-29T10:44:18.176 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feae6ffd640 1 -- 172.21.15.57:0/2325283958 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fead8005b50 con 0x7feaf80b17d0 2024-03-29T10:44:18.176 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feae6ffd640 1 -- 172.21.15.57:0/2325283958 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7feaf0001ad0 con 0x7feaf80b17d0 2024-03-29T10:44:18.176 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feae6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:18.176 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feae6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:18.176 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feae6ffd640 10 monclient(hunting): dump: 2024-03-29T10:44:18.176 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:18.176 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:18.176 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:18.176 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:18.177 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:18.177 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:18.177 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:18.177 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:18.177 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:18.177 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:18.179 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feae6ffd640 1 -- 172.21.15.57:0/2325283958 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 324634878 0 0) 0x7fead8005b50 con 0x7feaf80b17d0 2024-03-29T10:44:18.179 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feae6ffd640 1 -- 172.21.15.57:0/2325283958 >> v1:172.21.15.57:6790/0 conn(0x7feaf8142690 legacy=0x7feaf80b0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:18.179 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feae6ffd640 1 -- 172.21.15.57:0/2325283958 >> v1:172.21.15.179:6789/0 conn(0x7feaf81422b0 legacy=0x7feaf8130580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:18.179 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feae6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:44:18.179 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feae6ffd640 1 monclient: found mon.a 2024-03-29T10:44:18.179 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feae6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:18.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feae6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:44:18.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feae6ffd640 1 -- 172.21.15.57:0/2325283958 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf80ad240 con 0x7feaf80b17d0 2024-03-29T10:44:18.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feae6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:44:18.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feae6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:44:18.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feae6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:18.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feafc965640 5 monclient: authenticate success, global_id 4380 2024-03-29T10:44:18.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feafc965640 10 monclient: _renew_subs 2024-03-29T10:44:18.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feafc965640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:44:18.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feafc965640 1 -- 172.21.15.57:0/2325283958 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaf80abf20 con 0x7feaf80b17d0 2024-03-29T10:44:18.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feafc965640 10 client.4380.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:18.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feafc965640 10 monclient: _renew_subs 2024-03-29T10:44:18.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feafc965640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:44:18.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feafc965640 1 -- 172.21.15.57:0/2325283958 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feaf80abab0 con 0x7feaf80b17d0 2024-03-29T10:44:18.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.189+0000 7feafc965640 1 librados: init done 2024-03-29T10:44:18.180 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.191+0000 7feafc965640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:18.181 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.191+0000 7feafc965640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:18.181 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.191+0000 7feafc965640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:44:18.181 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.191+0000 7feafc965640 1 -- 172.21.15.57:0/2325283958 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaf80b43b0 con 0x7feaf80b17d0 2024-03-29T10:44:18.181 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.193+0000 7feae6ffd640 1 -- 172.21.15.57:0/2325283958 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feaf0001fc0 con 0x7feaf80b17d0 2024-03-29T10:44:18.181 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.193+0000 7feae6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:18.181 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.194+0000 7feae6ffd640 1 -- 172.21.15.57:0/2325283958 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7feaf0003bb0 con 0x7feaf80b17d0 2024-03-29T10:44:18.181 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.194+0000 7feae6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:18.181 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.194+0000 7feae6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:18.181 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.194+0000 7feae6ffd640 10 monclient: dump: 2024-03-29T10:44:18.181 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:18.181 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:18.181 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:18.181 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:18.181 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:18.182 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:18.182 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:18.182 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:18.182 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:18.182 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:18.182 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.194+0000 7feae7fff640 4 set_mon_vals no callback set 2024-03-29T10:44:18.183 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.195+0000 7feae6ffd640 1 -- 172.21.15.57:0/2325283958 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7feaf0019b80 con 0x7feaf80b17d0 2024-03-29T10:44:18.183 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.196+0000 7feae6ffd640 1 -- 172.21.15.57:0/2325283958 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7feaf0053700 con 0x7feaf80b17d0 2024-03-29T10:44:18.183 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.196+0000 7feae6ffd640 10 client.4380.objecter ms_dispatch 0x7feaf8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:18.183 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.196+0000 7feae6ffd640 3 client.4380.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:18.183 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.196+0000 7feae6ffd640 3 client.4380.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:18.183 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.196+0000 7feae6ffd640 20 client.4380.objecter dump_active .. 0 homeless 2024-03-29T10:44:18.183 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.197+0000 7feaf5d9b640 10 monclient: get_auth_request con 0x7fead806e240 auth_method 0 2024-03-29T10:44:18.184 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.197+0000 7feae6ffd640 10 client.4380.objecter ms_handle_connect 0x7fead806e240 2024-03-29T10:44:18.185 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.202+0000 7feae6ffd640 1 -- 172.21.15.57:0/2325283958 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feaf004f9a0 con 0x7feaf80b17d0 2024-03-29T10:44:18.186 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.202+0000 7feae6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:18.186 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.202+0000 7feae6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:18.526 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.542+0000 7feafc965640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:18.527 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.542+0000 7feafc965640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:18.527 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.542+0000 7feafc965640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:44:18.527 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.542+0000 7feafc965640 1 -- 172.21.15.57:0/2325283958 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feaf81328b0 con 0x7feaf80b17d0 2024-03-29T10:44:18.527 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.544+0000 7feae6ffd640 1 -- 172.21.15.57:0/2325283958 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7feaf004fb40 con 0x7feaf80b17d0 2024-03-29T10:44:18.527 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.544+0000 7feae6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:18.528 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.544+0000 7feae6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:18.528 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:44:18.528 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:18.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.547+0000 7feafc965640 10 librados: watch_flush enter 2024-03-29T10:44:18.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.548+0000 7feafc965640 10 librados: watch_flush exit 2024-03-29T10:44:18.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.548+0000 7feafc965640 20 client.4380.objecter shutdown clearing up homeless session... 2024-03-29T10:44:18.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.548+0000 7feafc965640 10 client.4380.objecter successfully canceled tick 2024-03-29T10:44:18.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.548+0000 7feafc965640 1 -- 172.21.15.57:0/2325283958 >> v1:172.21.15.179:6816/2698295936 conn(0x7fead806e240 legacy=0x7fead8070700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:18.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.548+0000 7feafc965640 10 monclient: shutdown 2024-03-29T10:44:18.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.548+0000 7feafc965640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:18.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.548+0000 7feafc965640 1 -- 172.21.15.57:0/2325283958 >> v1:172.21.15.57:6789/0 conn(0x7feaf80b17d0 legacy=0x7feaf812fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:18.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.548+0000 7feafc965640 1 -- 172.21.15.57:0/2325283958 shutdown_connections 2024-03-29T10:44:18.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.548+0000 7feafc965640 1 -- 172.21.15.57:0/2325283958 >> 172.21.15.57:0/2325283958 conn(0x7feaf8066a00 msgr2=0x7feaf80b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:18.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.548+0000 7feafc965640 1 -- 172.21.15.57:0/2325283958 shutdown_connections 2024-03-29T10:44:18.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.548+0000 7feafc965640 1 -- 172.21.15.57:0/2325283958 wait complete. 2024-03-29T10:44:18.533 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:18.548+0000 7feafc965640 1 librados: shutdown 2024-03-29T10:44:18.548 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:25.355 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:25.501 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.517+0000 7f9f8117b640 10 monclient: get_monmap_and_config 2024-03-29T10:44:25.501 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.517+0000 7f9f8117b640 10 monclient: build_initial_monmap 2024-03-29T10:44:25.501 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.517+0000 7f9f8117b640 10 monclient: monmap: 2024-03-29T10:44:25.501 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:44:25.501 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:25.501 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:44:25.518969+0000 2024-03-29T10:44:25.501 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:44:25.518969+0000 2024-03-29T10:44:25.501 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:25.501 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:25.502 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:44:25.502 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:44:25.502 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:44:25.502 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:25.503 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.518+0000 7f9f8117b640 1 Processor -- start 2024-03-29T10:44:25.503 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.518+0000 7f9f8117b640 1 -- start start 2024-03-29T10:44:25.503 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.518+0000 7f9f8117b640 10 monclient: init 2024-03-29T10:44:25.504 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.519+0000 7f9f8117b640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:25.504 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.519+0000 7f9f8117b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:25.504 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.519+0000 7f9f8117b640 10 monclient(hunting): picked mon.noname-a con 0x7f9f7c0b0770 addr v1:172.21.15.57:6789/0 2024-03-29T10:44:25.504 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.519+0000 7f9f8117b640 10 monclient(hunting): picked mon.noname-c con 0x7f9f7c14a700 addr v1:172.21.15.179:6789/0 2024-03-29T10:44:25.504 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.519+0000 7f9f8117b640 10 monclient(hunting): picked mon.noname-b con 0x7f9f7c0ac5a0 addr v1:172.21.15.57:6790/0 2024-03-29T10:44:25.504 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.519+0000 7f9f8117b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f7c0aba70 con 0x7f9f7c0b0770 2024-03-29T10:44:25.504 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.519+0000 7f9f8117b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f7c0aecb0 con 0x7f9f7c14a700 2024-03-29T10:44:25.504 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.519+0000 7f9f8117b640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f7c0af150 con 0x7f9f7c0ac5a0 2024-03-29T10:44:25.504 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.519+0000 7f9f8117b640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:25.504 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.519+0000 7f9f8117b640 10 monclient(hunting): authenticate will time out at 1217.696272s 2024-03-29T10:44:25.504 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.519+0000 7f9f7a59c640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f9f7c14a700 0x7f9f7c0abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:39262/0 (socket says 172.21.15.57:39262) 2024-03-29T10:44:25.504 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.519+0000 7f9f7a59c640 1 -- 172.21.15.57:0/579081040 learned_addr learned my addr 172.21.15.57:0/579081040 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:44:25.506 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.522+0000 7f9f79d9b640 1 -- 172.21.15.57:0/579081040 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4238245537 0 0) 0x7f9f7c0aecb0 con 0x7f9f7c14a700 2024-03-29T10:44:25.506 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.523+0000 7f9f79d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:44:25.506 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.523+0000 7f9f79d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:25.507 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.523+0000 7f9f79d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:25.507 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.523+0000 7f9f79d9b640 10 monclient(hunting): my global_id is 4390 2024-03-29T10:44:25.507 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.523+0000 7f9f79d9b640 1 -- 172.21.15.57:0/579081040 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f64003520 con 0x7f9f7c14a700 2024-03-29T10:44:25.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.523+0000 7f9f79d9b640 1 -- 172.21.15.57:0/579081040 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624991460 0 0) 0x7f9f7c0af150 con 0x7f9f7c0ac5a0 2024-03-29T10:44:25.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.523+0000 7f9f79d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:44:25.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.523+0000 7f9f79d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:25.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.523+0000 7f9f79d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:25.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.523+0000 7f9f79d9b640 10 monclient(hunting): my global_id is 4406 2024-03-29T10:44:25.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.523+0000 7f9f79d9b640 1 -- 172.21.15.57:0/579081040 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f7c0aecb0 con 0x7f9f7c0ac5a0 2024-03-29T10:44:25.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.523+0000 7f9f79d9b640 1 -- 172.21.15.57:0/579081040 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3229455514 0 0) 0x7f9f64003520 con 0x7f9f7c14a700 2024-03-29T10:44:25.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.523+0000 7f9f79d9b640 1 -- 172.21.15.57:0/579081040 >> v1:172.21.15.57:6790/0 conn(0x7f9f7c0ac5a0 legacy=0x7f9f7c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:25.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.523+0000 7f9f79d9b640 1 -- 172.21.15.57:0/579081040 >> v1:172.21.15.57:6789/0 conn(0x7f9f7c0b0770 legacy=0x7f9f7c149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:25.510 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.525+0000 7f9f79d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:44:25.510 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.525+0000 7f9f79d9b640 1 monclient: found mon.noname-c 2024-03-29T10:44:25.510 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.525+0000 7f9f79d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:25.510 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.525+0000 7f9f79d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:44:25.510 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.525+0000 7f9f79d9b640 1 -- 172.21.15.57:0/579081040 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f7c0ac9d0 con 0x7f9f7c14a700 2024-03-29T10:44:25.510 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.525+0000 7f9f79d9b640 10 monclient: _finish_auth 0 2024-03-29T10:44:25.510 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.525+0000 7f9f79d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:44:25.510 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.525+0000 7f9f79d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:25.511 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.525+0000 7f9f79d9b640 1 -- 172.21.15.57:0/579081040 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f9f700016b0 con 0x7f9f7c14a700 2024-03-29T10:44:25.511 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.525+0000 7f9f79d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:25.511 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.525+0000 7f9f79d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:25.511 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.525+0000 7f9f79d9b640 10 monclient: dump: 2024-03-29T10:44:25.511 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:25.511 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:25.511 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:25.511 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:25.511 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:25.511 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:25.511 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:25.511 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:25.511 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:25.511 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:25.512 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.526+0000 7f9f79d9b640 1 -- 172.21.15.57:0/579081040 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f70001810 con 0x7f9f7c14a700 2024-03-29T10:44:25.512 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.526+0000 7f9f79d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:25.512 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.526+0000 7f9f79d9b640 1 -- 172.21.15.57:0/579081040 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f9f70001c20 con 0x7f9f7c14a700 2024-03-29T10:44:25.512 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.526+0000 7f9f79d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:25.512 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.526+0000 7f9f79d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:25.512 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.526+0000 7f9f79d9b640 10 monclient: dump: 2024-03-29T10:44:25.512 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:25.512 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:25.512 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:25.512 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:25.512 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:25.512 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:25.512 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:25.512 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:25.512 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:25.513 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:25.514 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.526+0000 7f9f8117b640 5 monclient: authenticate success, global_id 4390 2024-03-29T10:44:25.514 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.526+0000 7f9f8117b640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:25.514 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.526+0000 7f9f8117b640 4 set_mon_vals no callback set 2024-03-29T10:44:25.514 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.527+0000 7f9f8117b640 10 monclient: shutdown 2024-03-29T10:44:25.514 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.527+0000 7f9f8117b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:25.514 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.527+0000 7f9f8117b640 1 -- 172.21.15.57:0/579081040 >> v1:172.21.15.179:6789/0 conn(0x7f9f7c14a700 legacy=0x7f9f7c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:25.514 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.527+0000 7f9f8117b640 1 -- 172.21.15.57:0/579081040 shutdown_connections 2024-03-29T10:44:25.514 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.527+0000 7f9f8117b640 1 -- 172.21.15.57:0/579081040 >> 172.21.15.57:0/579081040 conn(0x7f9f7c06b660 msgr2=0x7f9f7c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:25.514 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.527+0000 7f9f8117b640 1 -- 172.21.15.57:0/579081040 shutdown_connections 2024-03-29T10:44:25.515 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.528+0000 7f9f8117b640 1 -- 172.21.15.57:0/579081040 wait complete. 2024-03-29T10:44:25.515 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.529+0000 7f9f8117b640 10 monclient: build_initial_monmap 2024-03-29T10:44:25.515 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.529+0000 7f9f8117b640 10 monclient: monmap: 2024-03-29T10:44:25.515 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:44:25.515 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:25.515 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:44:25.515 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:44:25.515 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:25.515 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:25.515 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:44:25.515 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:44:25.515 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:44:25.515 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:25.515 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.529+0000 7f9f8117b640 1 librados: starting msgr at 2024-03-29T10:44:25.515 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.529+0000 7f9f8117b640 1 librados: starting objecter 2024-03-29T10:44:25.516 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.529+0000 7f9f8117b640 1 Processor -- start 2024-03-29T10:44:25.516 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.530+0000 7f9f8117b640 1 -- start start 2024-03-29T10:44:25.523 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.530+0000 7f9f8117b640 1 librados: setting wanted keys 2024-03-29T10:44:25.523 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.530+0000 7f9f8117b640 1 librados: calling monclient init 2024-03-29T10:44:25.523 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.530+0000 7f9f8117b640 10 monclient: init 2024-03-29T10:44:25.523 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.530+0000 7f9f8117b640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:25.523 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.530+0000 7f9f8117b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:44:25.523 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.531+0000 7f9f8117b640 10 monclient(hunting): picked mon.noname-b con 0x7f9f7c0ac5a0 addr v1:172.21.15.179:6789/0 2024-03-29T10:44:25.523 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.531+0000 7f9f8117b640 10 monclient(hunting): picked mon.noname-c con 0x7f9f7c0b0770 addr v1:172.21.15.57:6790/0 2024-03-29T10:44:25.523 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.531+0000 7f9f8117b640 10 monclient(hunting): picked mon.noname-a con 0x7f9f7c14a700 addr v1:172.21.15.57:6789/0 2024-03-29T10:44:25.524 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.531+0000 7f9f8117b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f7c11f590 con 0x7f9f7c14a700 2024-03-29T10:44:25.524 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.531+0000 7f9f8117b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f7c11d0d0 con 0x7f9f7c0ac5a0 2024-03-29T10:44:25.524 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.531+0000 7f9f8117b640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f7c1485d0 con 0x7f9f7c0b0770 2024-03-29T10:44:25.524 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.531+0000 7f9f8117b640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:25.524 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.531+0000 7f9f8117b640 10 monclient(hunting): authenticate will time out at 1217.708197s 2024-03-29T10:44:25.524 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.534+0000 7f9f7a59c640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f9f7c0b0770 0x7f9f7c146ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:34877/0 (socket says 172.21.15.57:34877) 2024-03-29T10:44:25.524 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.534+0000 7f9f7a59c640 1 -- 172.21.15.57:0/633559827 learned_addr learned my addr 172.21.15.57:0/633559827 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:44:25.524 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.534+0000 7f9f637fe640 10 client.?.objecter ms_handle_connect 0x7f9f7c0b0770 2024-03-29T10:44:25.524 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.534+0000 7f9f637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:25.524 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.534+0000 7f9f637fe640 1 -- 172.21.15.57:0/633559827 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 987605399 0 0) 0x7f9f7c1485d0 con 0x7f9f7c0b0770 2024-03-29T10:44:25.524 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.534+0000 7f9f637fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:25.524 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.535+0000 7f9f637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:25.524 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.535+0000 7f9f637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:25.524 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.535+0000 7f9f637fe640 10 monclient(hunting): my global_id is 4409 2024-03-29T10:44:25.524 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.535+0000 7f9f637fe640 1 -- 172.21.15.57:0/633559827 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f68004eb0 con 0x7f9f7c0b0770 2024-03-29T10:44:25.525 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.535+0000 7f9f637fe640 10 client.?.objecter ms_handle_connect 0x7f9f7c0ac5a0 2024-03-29T10:44:25.525 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.535+0000 7f9f637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:25.525 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.535+0000 7f9f637fe640 10 client.?.objecter ms_handle_connect 0x7f9f7c14a700 2024-03-29T10:44:25.525 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.535+0000 7f9f637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:25.525 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.535+0000 7f9f637fe640 1 -- 172.21.15.57:0/633559827 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878814620 0 0) 0x7f9f7c11f590 con 0x7f9f7c14a700 2024-03-29T10:44:25.525 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.535+0000 7f9f637fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:25.525 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.535+0000 7f9f637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:25.525 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.535+0000 7f9f637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:25.525 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.535+0000 7f9f637fe640 10 monclient(hunting): my global_id is 4383 2024-03-29T10:44:25.525 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.535+0000 7f9f637fe640 1 -- 172.21.15.57:0/633559827 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f7c1485d0 con 0x7f9f7c14a700 2024-03-29T10:44:25.525 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.536+0000 7f9f637fe640 1 -- 172.21.15.57:0/633559827 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617551720 0 0) 0x7f9f7c11d0d0 con 0x7f9f7c0ac5a0 2024-03-29T10:44:25.525 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.536+0000 7f9f637fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:25.525 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.536+0000 7f9f637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:25.525 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.536+0000 7f9f637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:25.526 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.536+0000 7f9f637fe640 10 monclient(hunting): my global_id is 4393 2024-03-29T10:44:25.526 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.536+0000 7f9f637fe640 1 -- 172.21.15.57:0/633559827 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f7c11f590 con 0x7f9f7c0ac5a0 2024-03-29T10:44:25.526 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.536+0000 7f9f637fe640 1 -- 172.21.15.57:0/633559827 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3889207455 0 0) 0x7f9f7c1485d0 con 0x7f9f7c14a700 2024-03-29T10:44:25.526 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.536+0000 7f9f637fe640 1 -- 172.21.15.57:0/633559827 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f7c11d0d0 con 0x7f9f7c14a700 2024-03-29T10:44:25.526 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.536+0000 7f9f637fe640 1 -- 172.21.15.57:0/633559827 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f9f6c0018b0 con 0x7f9f7c14a700 2024-03-29T10:44:25.526 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.536+0000 7f9f637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:25.526 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.536+0000 7f9f637fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:25.526 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.536+0000 7f9f637fe640 10 monclient(hunting): dump: 2024-03-29T10:44:25.526 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:25.526 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:25.526 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:25.526 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:25.526 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:25.526 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:25.526 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:25.527 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:25.527 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:25.527 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:25.527 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.536+0000 7f9f637fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:25.527 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.536+0000 7f9f637fe640 1 -- 172.21.15.57:0/633559827 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1288635580 0 0) 0x7f9f7c11d0d0 con 0x7f9f7c14a700 2024-03-29T10:44:25.527 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.537+0000 7f9f637fe640 1 -- 172.21.15.57:0/633559827 >> v1:172.21.15.57:6790/0 conn(0x7f9f7c0b0770 legacy=0x7f9f7c146ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:25.527 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.537+0000 7f9f637fe640 1 -- 172.21.15.57:0/633559827 >> v1:172.21.15.179:6789/0 conn(0x7f9f7c0ac5a0 legacy=0x7f9f7c11da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:25.527 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.537+0000 7f9f637fe640 10 monclient: _finish_hunting 0 2024-03-29T10:44:25.527 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.537+0000 7f9f637fe640 1 monclient: found mon.a 2024-03-29T10:44:25.527 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.537+0000 7f9f637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:25.527 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.537+0000 7f9f637fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:44:25.527 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.537+0000 7f9f637fe640 1 -- 172.21.15.57:0/633559827 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f7c11e1a0 con 0x7f9f7c14a700 2024-03-29T10:44:25.527 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.537+0000 7f9f637fe640 10 monclient: _finish_auth 0 2024-03-29T10:44:25.527 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.537+0000 7f9f637fe640 10 monclient: _check_auth_tickets 2024-03-29T10:44:25.527 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.537+0000 7f9f637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:25.528 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.537+0000 7f9f637fe640 1 -- 172.21.15.57:0/633559827 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f6c003610 con 0x7f9f7c14a700 2024-03-29T10:44:25.528 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.537+0000 7f9f637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:25.528 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.537+0000 7f9f637fe640 1 -- 172.21.15.57:0/633559827 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f9f6c003920 con 0x7f9f7c14a700 2024-03-29T10:44:25.528 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.537+0000 7f9f637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:25.528 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.537+0000 7f9f637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:25.528 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.537+0000 7f9f637fe640 10 monclient: dump: 2024-03-29T10:44:25.528 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:25.528 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:25.528 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:25.528 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:25.528 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:25.528 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:25.528 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:25.528 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:25.528 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:25.529 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:25.529 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.537+0000 7f9f7959a640 4 set_mon_vals no callback set 2024-03-29T10:44:25.529 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.538+0000 7f9f8117b640 5 monclient: authenticate success, global_id 4383 2024-03-29T10:44:25.529 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.538+0000 7f9f8117b640 10 monclient: _renew_subs 2024-03-29T10:44:25.529 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.538+0000 7f9f8117b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:44:25.529 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.538+0000 7f9f8117b640 1 -- 172.21.15.57:0/633559827 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f7c11d5f0 con 0x7f9f7c14a700 2024-03-29T10:44:25.529 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.539+0000 7f9f8117b640 10 client.4383.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:25.529 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.539+0000 7f9f8117b640 10 monclient: _renew_subs 2024-03-29T10:44:25.529 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.539+0000 7f9f8117b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:44:25.529 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.539+0000 7f9f8117b640 1 -- 172.21.15.57:0/633559827 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f7c147d80 con 0x7f9f7c14a700 2024-03-29T10:44:25.529 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.539+0000 7f9f8117b640 1 librados: init done 2024-03-29T10:44:25.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.540+0000 7f9f8117b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:25.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.540+0000 7f9f8117b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:25.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.540+0000 7f9f8117b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:44:25.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.540+0000 7f9f8117b640 1 -- 172.21.15.57:0/633559827 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f7c11e390 con 0x7f9f7c14a700 2024-03-29T10:44:25.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.541+0000 7f9f637fe640 1 -- 172.21.15.57:0/633559827 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f9f6c0198f0 con 0x7f9f7c14a700 2024-03-29T10:44:25.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.542+0000 7f9f637fe640 1 -- 172.21.15.57:0/633559827 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f9f6c0837d0 con 0x7f9f7c14a700 2024-03-29T10:44:25.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.543+0000 7f9f637fe640 10 client.4383.objecter ms_dispatch 0x7f9f7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:25.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.543+0000 7f9f637fe640 3 client.4383.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:25.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.543+0000 7f9f637fe640 3 client.4383.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:25.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.543+0000 7f9f637fe640 20 client.4383.objecter dump_active .. 0 homeless 2024-03-29T10:44:25.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.544+0000 7f9f7ad9d640 10 monclient: get_auth_request con 0x7f9f6806df50 auth_method 0 2024-03-29T10:44:25.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.544+0000 7f9f637fe640 10 client.4383.objecter ms_handle_connect 0x7f9f6806df50 2024-03-29T10:44:25.533 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.550+0000 7f9f637fe640 1 -- 172.21.15.57:0/633559827 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9f6c04f0b0 con 0x7f9f7c14a700 2024-03-29T10:44:25.533 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.550+0000 7f9f637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:25.534 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.550+0000 7f9f637fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:25.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.869+0000 7f9f8117b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:25.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.869+0000 7f9f8117b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:25.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.869+0000 7f9f8117b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:44:25.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.869+0000 7f9f8117b640 1 -- 172.21.15.57:0/633559827 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f7c0ab560 con 0x7f9f7c14a700 2024-03-29T10:44:25.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.870+0000 7f9f637fe640 1 -- 172.21.15.57:0/633559827 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f9f6c050fe0 con 0x7f9f7c14a700 2024-03-29T10:44:25.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.870+0000 7f9f637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:25.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.870+0000 7f9f637fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:25.855 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:44:25.855 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:25.859 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.875+0000 7f9f8117b640 10 librados: watch_flush enter 2024-03-29T10:44:25.859 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.875+0000 7f9f8117b640 10 librados: watch_flush exit 2024-03-29T10:44:25.859 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.875+0000 7f9f8117b640 20 client.4383.objecter shutdown clearing up homeless session... 2024-03-29T10:44:25.859 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.875+0000 7f9f8117b640 10 client.4383.objecter successfully canceled tick 2024-03-29T10:44:25.859 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.875+0000 7f9f8117b640 1 -- 172.21.15.57:0/633559827 >> v1:172.21.15.179:6816/2698295936 conn(0x7f9f6806df50 legacy=0x7f9f68070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:25.859 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.875+0000 7f9f8117b640 10 monclient: shutdown 2024-03-29T10:44:25.859 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.875+0000 7f9f8117b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:25.859 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.875+0000 7f9f8117b640 1 -- 172.21.15.57:0/633559827 >> v1:172.21.15.57:6789/0 conn(0x7f9f7c14a700 legacy=0x7f9f7c1ef690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:25.859 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.875+0000 7f9f8117b640 1 -- 172.21.15.57:0/633559827 shutdown_connections 2024-03-29T10:44:25.860 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.875+0000 7f9f8117b640 1 -- 172.21.15.57:0/633559827 >> 172.21.15.57:0/633559827 conn(0x7f9f7c06b660 msgr2=0x7f9f7c0abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:25.860 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.876+0000 7f9f8117b640 1 -- 172.21.15.57:0/633559827 shutdown_connections 2024-03-29T10:44:25.860 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.876+0000 7f9f8117b640 1 -- 172.21.15.57:0/633559827 wait complete. 2024-03-29T10:44:25.860 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:25.876+0000 7f9f8117b640 1 librados: shutdown 2024-03-29T10:44:25.876 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:32.683 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:32.843 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.859+0000 7f7164255640 10 monclient: get_monmap_and_config 2024-03-29T10:44:32.844 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.859+0000 7f7164255640 10 monclient: build_initial_monmap 2024-03-29T10:44:32.844 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.859+0000 7f7164255640 10 monclient: monmap: 2024-03-29T10:44:32.844 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:44:32.844 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:32.844 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:44:32.861160+0000 2024-03-29T10:44:32.844 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:44:32.861160+0000 2024-03-29T10:44:32.844 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:32.844 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:32.844 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:44:32.844 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:44:32.844 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:44:32.844 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:32.846 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.860+0000 7f7164255640 1 Processor -- start 2024-03-29T10:44:32.846 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.860+0000 7f7164255640 1 -- start start 2024-03-29T10:44:32.846 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.860+0000 7f7164255640 10 monclient: init 2024-03-29T10:44:32.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.861+0000 7f7164255640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:32.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.861+0000 7f7164255640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:32.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.861+0000 7f7164255640 10 monclient(hunting): picked mon.noname-a con 0x7f715c0b0860 addr v1:172.21.15.57:6789/0 2024-03-29T10:44:32.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.861+0000 7f7164255640 10 monclient(hunting): picked mon.noname-c con 0x7f715c13ef30 addr v1:172.21.15.179:6789/0 2024-03-29T10:44:32.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.861+0000 7f7164255640 10 monclient(hunting): picked mon.noname-b con 0x7f715c0af8e0 addr v1:172.21.15.57:6790/0 2024-03-29T10:44:32.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.861+0000 7f7164255640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c0b0c40 con 0x7f715c0b0860 2024-03-29T10:44:32.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.861+0000 7f7164255640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c0aed10 con 0x7f715c13ef30 2024-03-29T10:44:32.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.861+0000 7f7164255640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c0afd50 con 0x7f715c0af8e0 2024-03-29T10:44:32.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.861+0000 7f7164255640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:32.847 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.861+0000 7f7164255640 10 monclient(hunting): authenticate will time out at 1225.038844s 2024-03-29T10:44:32.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.867+0000 7f71627f2640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f715c0af8e0 0x7f715c149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:57721/0 (socket says 172.21.15.57:57721) 2024-03-29T10:44:32.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.867+0000 7f71627f2640 1 -- 172.21.15.57:0/2292025864 learned_addr learned my addr 172.21.15.57:0/2292025864 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:44:32.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.867+0000 7f7160fef640 1 -- 172.21.15.57:0/2292025864 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1035946532 0 0) 0x7f715c0b0c40 con 0x7f715c0b0860 2024-03-29T10:44:32.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.867+0000 7f7160fef640 10 monclient(hunting): _negotiate 2024-03-29T10:44:32.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.867+0000 7f7160fef640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:32.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.867+0000 7f7160fef640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:32.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.867+0000 7f7160fef640 10 monclient(hunting): my global_id is 4386 2024-03-29T10:44:32.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.868+0000 7f7160fef640 1 -- 172.21.15.57:0/2292025864 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7148003520 con 0x7f715c0b0860 2024-03-29T10:44:32.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.868+0000 7f7160fef640 1 -- 172.21.15.57:0/2292025864 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3836425707 0 0) 0x7f7148003520 con 0x7f715c0b0860 2024-03-29T10:44:32.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.868+0000 7f7160fef640 1 -- 172.21.15.57:0/2292025864 >> v1:172.21.15.57:6790/0 conn(0x7f715c0af8e0 legacy=0x7f715c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:32.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.868+0000 7f7160fef640 1 -- 172.21.15.57:0/2292025864 >> v1:172.21.15.179:6789/0 conn(0x7f715c13ef30 legacy=0x7f715c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:44:32.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.868+0000 7f7160fef640 10 monclient: _finish_hunting 0 2024-03-29T10:44:32.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.868+0000 7f7160fef640 1 monclient: found mon.noname-a 2024-03-29T10:44:32.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.868+0000 7f7160fef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:32.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.868+0000 7f7160fef640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:44:32.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.868+0000 7f7160fef640 1 -- 172.21.15.57:0/2292025864 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715c0ab1b0 con 0x7f715c0b0860 2024-03-29T10:44:32.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.868+0000 7f7160fef640 10 monclient: _finish_auth 0 2024-03-29T10:44:32.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.868+0000 7f7160fef640 10 monclient: _check_auth_tickets 2024-03-29T10:44:32.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.868+0000 7f7160fef640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:32.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.869+0000 7f7164255640 5 monclient: authenticate success, global_id 4386 2024-03-29T10:44:32.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.869+0000 7f7164255640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:32.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.869+0000 7f7160fef640 1 -- 172.21.15.57:0/2292025864 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f7154001a10 con 0x7f715c0b0860 2024-03-29T10:44:32.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.869+0000 7f7160fef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:32.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.869+0000 7f7160fef640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:32.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.869+0000 7f7160fef640 10 monclient: dump: 2024-03-29T10:44:32.855 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:32.855 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:32.856 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:32.856 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:32.856 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:32.856 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:32.856 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:32.856 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:32.856 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:32.856 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:32.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.869+0000 7f7164255640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:32.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.869+0000 7f7160fef640 1 -- 172.21.15.57:0/2292025864 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7154001ca0 con 0x7f715c0b0860 2024-03-29T10:44:32.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.869+0000 7f7160fef640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:32.857 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.869+0000 7f7164255640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:32.857 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.869+0000 7f7164255640 4 set_mon_vals no callback set 2024-03-29T10:44:32.857 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.870+0000 7f7164255640 10 monclient: shutdown 2024-03-29T10:44:32.857 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.870+0000 7f7164255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:32.857 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.870+0000 7f7164255640 1 -- 172.21.15.57:0/2292025864 >> v1:172.21.15.57:6789/0 conn(0x7f715c0b0860 legacy=0x7f715c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:32.857 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.870+0000 7f7160fef640 1 -- 172.21.15.57:0/2292025864 <== mon.0 v1:172.21.15.57:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f7154002030 con 0x7f715c0b0860 2024-03-29T10:44:32.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.870+0000 7f7160fef640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:44:32.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.871+0000 7f7164255640 1 -- 172.21.15.57:0/2292025864 shutdown_connections 2024-03-29T10:44:32.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.871+0000 7f7164255640 1 -- 172.21.15.57:0/2292025864 >> 172.21.15.57:0/2292025864 conn(0x7f715c06b660 msgr2=0x7f715c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:32.860 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.872+0000 7f7164255640 1 -- 172.21.15.57:0/2292025864 shutdown_connections 2024-03-29T10:44:32.861 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.872+0000 7f7164255640 1 -- 172.21.15.57:0/2292025864 wait complete. 2024-03-29T10:44:32.861 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.873+0000 7f7164255640 10 monclient: build_initial_monmap 2024-03-29T10:44:32.861 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.873+0000 7f7164255640 10 monclient: monmap: 2024-03-29T10:44:32.861 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:44:32.861 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:32.861 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:44:32.861 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:44:32.861 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:32.861 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:32.861 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:44:32.861 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:44:32.861 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:44:32.862 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:32.862 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.874+0000 7f7164255640 1 librados: starting msgr at 2024-03-29T10:44:32.862 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.874+0000 7f7164255640 1 librados: starting objecter 2024-03-29T10:44:32.862 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.874+0000 7f7164255640 1 Processor -- start 2024-03-29T10:44:32.862 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.874+0000 7f7164255640 1 -- start start 2024-03-29T10:44:32.862 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.874+0000 7f7164255640 1 librados: setting wanted keys 2024-03-29T10:44:32.862 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.874+0000 7f7164255640 1 librados: calling monclient init 2024-03-29T10:44:32.862 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.874+0000 7f7164255640 10 monclient: init 2024-03-29T10:44:32.862 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.875+0000 7f7164255640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:32.862 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.875+0000 7f7164255640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:44:32.862 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.875+0000 7f7164255640 10 monclient(hunting): picked mon.noname-b con 0x7f715c0af8e0 addr v1:172.21.15.179:6789/0 2024-03-29T10:44:32.863 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.875+0000 7f7164255640 10 monclient(hunting): picked mon.noname-a con 0x7f715c0b0860 addr v1:172.21.15.57:6789/0 2024-03-29T10:44:32.863 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.875+0000 7f7164255640 10 monclient(hunting): picked mon.noname-c con 0x7f715c13ef30 addr v1:172.21.15.57:6790/0 2024-03-29T10:44:32.863 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.875+0000 7f7164255640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c06fb60 con 0x7f715c0b0860 2024-03-29T10:44:32.863 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.875+0000 7f7164255640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c070a00 con 0x7f715c0af8e0 2024-03-29T10:44:32.863 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.875+0000 7f7164255640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f715c070ce0 con 0x7f715c13ef30 2024-03-29T10:44:32.863 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.875+0000 7f7164255640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:32.863 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.875+0000 7f7164255640 10 monclient(hunting): authenticate will time out at 1225.052484s 2024-03-29T10:44:32.864 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.880+0000 7f71627f2640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f715c13ef30 0x7f715c1ffef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:57825/0 (socket says 172.21.15.57:57825) 2024-03-29T10:44:32.864 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.880+0000 7f71627f2640 1 -- 172.21.15.57:0/3541225719 learned_addr learned my addr 172.21.15.57:0/3541225719 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:44:32.868 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.884+0000 7f71527fc640 10 client.?.objecter ms_handle_connect 0x7f715c13ef30 2024-03-29T10:44:32.868 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.884+0000 7f71527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:32.868 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.884+0000 7f71527fc640 10 client.?.objecter ms_handle_connect 0x7f715c0af8e0 2024-03-29T10:44:32.868 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.884+0000 7f71527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:32.868 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.884+0000 7f71527fc640 10 client.?.objecter ms_handle_connect 0x7f715c0b0860 2024-03-29T10:44:32.868 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.884+0000 7f71527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:32.868 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.884+0000 7f71527fc640 1 -- 172.21.15.57:0/3541225719 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820194888 0 0) 0x7f715c070ce0 con 0x7f715c13ef30 2024-03-29T10:44:32.868 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.884+0000 7f71527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:32.868 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.884+0000 7f71527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:32.869 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.884+0000 7f71527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:32.869 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.884+0000 7f71527fc640 10 monclient(hunting): my global_id is 4415 2024-03-29T10:44:32.869 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.884+0000 7f71527fc640 1 -- 172.21.15.57:0/3541225719 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7140004eb0 con 0x7f715c13ef30 2024-03-29T10:44:32.869 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.884+0000 7f71527fc640 1 -- 172.21.15.57:0/3541225719 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177464100 0 0) 0x7f715c06fb60 con 0x7f715c0b0860 2024-03-29T10:44:32.869 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.884+0000 7f71527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:32.869 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.884+0000 7f71527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:32.869 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.884+0000 7f71527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:32.869 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.884+0000 7f71527fc640 10 monclient(hunting): my global_id is 4389 2024-03-29T10:44:32.869 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.884+0000 7f71527fc640 1 -- 172.21.15.57:0/3541225719 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f715c070ce0 con 0x7f715c0b0860 2024-03-29T10:44:32.870 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.884+0000 7f71527fc640 1 -- 172.21.15.57:0/3541225719 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476450705 0 0) 0x7f715c070a00 con 0x7f715c0af8e0 2024-03-29T10:44:32.870 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.884+0000 7f71527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:32.870 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.884+0000 7f71527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:32.871 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.884+0000 7f71527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:32.871 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.884+0000 7f71527fc640 10 monclient(hunting): my global_id is 4396 2024-03-29T10:44:32.871 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.884+0000 7f71527fc640 1 -- 172.21.15.57:0/3541225719 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f715c06fb60 con 0x7f715c0af8e0 2024-03-29T10:44:32.871 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.885+0000 7f71527fc640 1 -- 172.21.15.57:0/3541225719 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4188508103 0 0) 0x7f7140004eb0 con 0x7f715c13ef30 2024-03-29T10:44:32.871 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.885+0000 7f71527fc640 1 -- 172.21.15.57:0/3541225719 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f715c070a00 con 0x7f715c13ef30 2024-03-29T10:44:32.871 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.885+0000 7f71527fc640 1 -- 172.21.15.57:0/3541225719 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f714c001740 con 0x7f715c13ef30 2024-03-29T10:44:32.871 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.885+0000 7f71527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:32.871 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.885+0000 7f71527fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:32.871 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.885+0000 7f71527fc640 10 monclient(hunting): dump: 2024-03-29T10:44:32.872 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:32.872 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:32.872 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:32.872 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:32.872 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:32.872 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:32.872 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:32.872 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:32.872 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:32.872 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:32.873 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.885+0000 7f71527fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:32.873 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.886+0000 7f71527fc640 1 -- 172.21.15.57:0/3541225719 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3568392704 0 0) 0x7f715c06fb60 con 0x7f715c0af8e0 2024-03-29T10:44:32.873 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.886+0000 7f71527fc640 1 -- 172.21.15.57:0/3541225719 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7140004eb0 con 0x7f715c0af8e0 2024-03-29T10:44:32.873 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.886+0000 7f71527fc640 1 -- 172.21.15.57:0/3541225719 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f71540018d0 con 0x7f715c0af8e0 2024-03-29T10:44:32.873 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.886+0000 7f71527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:32.873 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.886+0000 7f71527fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:32.873 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.886+0000 7f71527fc640 10 monclient(hunting): dump: 2024-03-29T10:44:32.873 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:32.873 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:32.873 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:32.874 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:32.874 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:32.874 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:32.874 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:32.874 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:32.874 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:32.874 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:32.874 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.886+0000 7f71527fc640 1 -- 172.21.15.57:0/3541225719 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3002827955 0 0) 0x7f715c070a00 con 0x7f715c13ef30 2024-03-29T10:44:32.874 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.886+0000 7f71527fc640 1 -- 172.21.15.57:0/3541225719 >> v1:172.21.15.179:6789/0 conn(0x7f715c0af8e0 legacy=0x7f715c06e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:32.874 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.886+0000 7f71527fc640 1 -- 172.21.15.57:0/3541225719 >> v1:172.21.15.57:6789/0 conn(0x7f715c0b0860 legacy=0x7f715c071100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:32.874 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.886+0000 7f71527fc640 10 monclient: _finish_hunting 0 2024-03-29T10:44:32.874 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.886+0000 7f71527fc640 1 monclient: found mon.c 2024-03-29T10:44:32.874 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.886+0000 7f71527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:32.874 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.886+0000 7f71527fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:44:32.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.886+0000 7f71527fc640 1 -- 172.21.15.57:0/3541225719 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715c06f640 con 0x7f715c13ef30 2024-03-29T10:44:32.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.886+0000 7f71527fc640 10 monclient: _finish_auth 0 2024-03-29T10:44:32.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.886+0000 7f71527fc640 10 monclient: _check_auth_tickets 2024-03-29T10:44:32.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.886+0000 7f71527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:32.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.886+0000 7f7164255640 5 monclient: authenticate success, global_id 4415 2024-03-29T10:44:32.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.886+0000 7f7164255640 10 monclient: _renew_subs 2024-03-29T10:44:32.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.886+0000 7f7164255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:44:32.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.886+0000 7f7164255640 1 -- 172.21.15.57:0/3541225719 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f715c06f1b0 con 0x7f715c13ef30 2024-03-29T10:44:32.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.887+0000 7f7164255640 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:32.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.887+0000 7f7164255640 10 monclient: _renew_subs 2024-03-29T10:44:32.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.887+0000 7f7164255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:44:32.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.887+0000 7f7164255640 1 -- 172.21.15.57:0/3541225719 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f715c0719f0 con 0x7f715c13ef30 2024-03-29T10:44:32.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.887+0000 7f7164255640 1 librados: init done 2024-03-29T10:44:32.876 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.888+0000 7f7164255640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:32.880 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.888+0000 7f7164255640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:32.880 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.888+0000 7f7164255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:44:32.880 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.888+0000 7f7164255640 1 -- 172.21.15.57:0/3541225719 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f715c075f10 con 0x7f715c13ef30 2024-03-29T10:44:32.880 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.889+0000 7f71527fc640 1 -- 172.21.15.57:0/3541225719 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f714c001c70 con 0x7f715c13ef30 2024-03-29T10:44:32.880 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.889+0000 7f71527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:32.880 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.889+0000 7f71527fc640 1 -- 172.21.15.57:0/3541225719 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f714c004ab0 con 0x7f715c13ef30 2024-03-29T10:44:32.880 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.889+0000 7f71527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:32.880 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.889+0000 7f71527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:32.880 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.889+0000 7f71527fc640 10 monclient: dump: 2024-03-29T10:44:32.880 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:32.880 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:32.880 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:32.880 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:32.881 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:32.881 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:32.881 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:32.881 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:32.881 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:32.881 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:32.881 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.889+0000 7f71527fc640 1 -- 172.21.15.57:0/3541225719 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f714c01aa80 con 0x7f715c13ef30 2024-03-29T10:44:32.881 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.890+0000 7f71527fc640 1 -- 172.21.15.57:0/3541225719 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f714c084ad0 con 0x7f715c13ef30 2024-03-29T10:44:32.881 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.890+0000 7f71527fc640 10 client.4415.objecter ms_dispatch 0x7f715c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:32.881 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.890+0000 7f71527fc640 3 client.4415.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:32.881 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.890+0000 7f71527fc640 3 client.4415.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:32.881 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.890+0000 7f71527fc640 20 client.4415.objecter dump_active .. 0 homeless 2024-03-29T10:44:32.881 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.890+0000 7f7153fff640 4 set_mon_vals no callback set 2024-03-29T10:44:32.883 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.899+0000 7f7161ff1640 10 monclient: get_auth_request con 0x7f714006df20 auth_method 0 2024-03-29T10:44:32.883 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.899+0000 7f71527fc640 1 -- 172.21.15.57:0/3541225719 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f714c050430 con 0x7f715c13ef30 2024-03-29T10:44:32.883 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.899+0000 7f71527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:32.883 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.899+0000 7f71527fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:32.883 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:32.899+0000 7f71527fc640 10 client.4415.objecter ms_handle_connect 0x7f714006df20 2024-03-29T10:44:33.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:33.221+0000 7f7164255640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:33.206 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:33.221+0000 7f7164255640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:33.206 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:33.221+0000 7f7164255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:44:33.206 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:33.221+0000 7f7164255640 1 -- 172.21.15.57:0/3541225719 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f715c06f7d0 con 0x7f715c13ef30 2024-03-29T10:44:33.206 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:33.222+0000 7f71527fc640 1 -- 172.21.15.57:0/3541225719 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f714c052360 con 0x7f715c13ef30 2024-03-29T10:44:33.206 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:33.222+0000 7f71527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:33.206 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:33.222+0000 7f71527fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:33.209 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:44:33.209 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:33.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:33.227+0000 7f7164255640 10 librados: watch_flush enter 2024-03-29T10:44:33.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:33.229+0000 7f7164255640 10 librados: watch_flush exit 2024-03-29T10:44:33.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:33.229+0000 7f7164255640 20 client.4415.objecter shutdown clearing up homeless session... 2024-03-29T10:44:33.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:33.229+0000 7f7164255640 10 client.4415.objecter successfully canceled tick 2024-03-29T10:44:33.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:33.230+0000 7f7164255640 1 -- 172.21.15.57:0/3541225719 >> v1:172.21.15.179:6816/2698295936 conn(0x7f714006df20 legacy=0x7f71400703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:33.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:33.230+0000 7f7164255640 10 monclient: shutdown 2024-03-29T10:44:33.214 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:33.230+0000 7f7164255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:33.214 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:33.230+0000 7f7164255640 1 -- 172.21.15.57:0/3541225719 >> v1:172.21.15.57:6790/0 conn(0x7f715c13ef30 legacy=0x7f715c1ffef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:33.214 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:33.230+0000 7f7164255640 1 -- 172.21.15.57:0/3541225719 shutdown_connections 2024-03-29T10:44:33.215 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:33.230+0000 7f7164255640 1 -- 172.21.15.57:0/3541225719 >> 172.21.15.57:0/3541225719 conn(0x7f715c06b660 msgr2=0x7f715c13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:33.215 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:33.230+0000 7f7164255640 1 -- 172.21.15.57:0/3541225719 shutdown_connections 2024-03-29T10:44:33.215 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:33.230+0000 7f7164255640 1 -- 172.21.15.57:0/3541225719 wait complete. 2024-03-29T10:44:33.215 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:33.230+0000 7f7164255640 1 librados: shutdown 2024-03-29T10:44:33.230 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:40.036 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:40.185 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.202+0000 7f1ebe94d640 10 monclient: get_monmap_and_config 2024-03-29T10:44:40.185 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.202+0000 7f1ebe94d640 10 monclient: build_initial_monmap 2024-03-29T10:44:40.185 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.202+0000 7f1ebe94d640 10 monclient: monmap: 2024-03-29T10:44:40.185 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:44:40.185 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:40.185 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:44:40.203072+0000 2024-03-29T10:44:40.185 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:44:40.203072+0000 2024-03-29T10:44:40.185 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:40.185 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:40.185 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:44:40.185 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:44:40.186 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:44:40.186 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:40.186 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.203+0000 7f1ebe94d640 1 Processor -- start 2024-03-29T10:44:40.186 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.203+0000 7f1ebe94d640 1 -- start start 2024-03-29T10:44:40.186 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.203+0000 7f1ebe94d640 10 monclient: init 2024-03-29T10:44:40.187 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.204+0000 7f1ebe94d640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:40.187 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.204+0000 7f1ebe94d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:40.191 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.208+0000 7f1ebe94d640 10 monclient(hunting): picked mon.noname-a con 0x7f1eb81457f0 addr v1:172.21.15.57:6789/0 2024-03-29T10:44:40.191 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.208+0000 7f1ebe94d640 10 monclient(hunting): picked mon.noname-c con 0x7f1eb813f650 addr v1:172.21.15.179:6789/0 2024-03-29T10:44:40.191 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.209+0000 7f1ebe94d640 10 monclient(hunting): picked mon.noname-b con 0x7f1eb80a9d70 addr v1:172.21.15.57:6790/0 2024-03-29T10:44:40.191 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.209+0000 7f1ebe94d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb80aaef0 con 0x7f1eb81457f0 2024-03-29T10:44:40.192 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.209+0000 7f1ebe94d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb8145bd0 con 0x7f1eb813f650 2024-03-29T10:44:40.192 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.209+0000 7f1ebe94d640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb8145eb0 con 0x7f1eb80a9d70 2024-03-29T10:44:40.192 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.209+0000 7f1ebe94d640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:40.192 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.209+0000 7f1ebe94d640 10 monclient(hunting): authenticate will time out at 1232.385456s 2024-03-29T10:44:40.194 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.211+0000 7f1eb7fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1eb81457f0 0x7f1eb813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:50183/0 (socket says 172.21.15.57:50183) 2024-03-29T10:44:40.194 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.211+0000 7f1eb7fff640 1 -- 172.21.15.57:0/3500108898 learned_addr learned my addr 172.21.15.57:0/3500108898 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:44:40.197 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.214+0000 7f1eb6ffd640 1 -- 172.21.15.57:0/3500108898 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1665133808 0 0) 0x7f1eb8145eb0 con 0x7f1eb80a9d70 2024-03-29T10:44:40.197 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.214+0000 7f1eb6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:40.197 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.214+0000 7f1eb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:40.197 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.214+0000 7f1eb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:40.197 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.214+0000 7f1eb6ffd640 10 monclient(hunting): my global_id is 4418 2024-03-29T10:44:40.197 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.214+0000 7f1eb6ffd640 1 -- 172.21.15.57:0/3500108898 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea0003520 con 0x7f1eb80a9d70 2024-03-29T10:44:40.197 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 1 -- 172.21.15.57:0/3500108898 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060362979 0 0) 0x7f1eb80aaef0 con 0x7f1eb81457f0 2024-03-29T10:44:40.197 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:40.197 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:40.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:40.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 10 monclient(hunting): my global_id is 4392 2024-03-29T10:44:40.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 1 -- 172.21.15.57:0/3500108898 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1eb8145eb0 con 0x7f1eb81457f0 2024-03-29T10:44:40.199 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 1 -- 172.21.15.57:0/3500108898 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 400856773 0 0) 0x7f1eb8145eb0 con 0x7f1eb81457f0 2024-03-29T10:44:40.199 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 1 -- 172.21.15.57:0/3500108898 >> v1:172.21.15.57:6790/0 conn(0x7f1eb80a9d70 legacy=0x7f1eb814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:40.199 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 1 -- 172.21.15.57:0/3500108898 >> v1:172.21.15.179:6789/0 conn(0x7f1eb813f650 legacy=0x7f1eb80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:44:40.199 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:44:40.199 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 1 monclient: found mon.noname-a 2024-03-29T10:44:40.199 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:40.199 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:44:40.199 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 1 -- 172.21.15.57:0/3500108898 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb8146340 con 0x7f1eb81457f0 2024-03-29T10:44:40.199 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:44:40.200 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:44:40.200 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:40.200 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 1 -- 172.21.15.57:0/3500108898 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1ea8001670 con 0x7f1eb81457f0 2024-03-29T10:44:40.200 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:40.200 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:40.200 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 10 monclient: dump: 2024-03-29T10:44:40.200 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:40.200 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:40.200 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:40.200 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:40.200 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:40.200 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:40.200 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:40.200 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:40.200 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:40.201 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:40.201 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1ebe94d640 5 monclient: authenticate success, global_id 4392 2024-03-29T10:44:40.201 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1ebe94d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:40.201 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 1 -- 172.21.15.57:0/3500108898 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ea8001890 con 0x7f1eb81457f0 2024-03-29T10:44:40.201 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:40.201 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 1 -- 172.21.15.57:0/3500108898 <== mon.0 v1:172.21.15.57:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1ea8001c20 con 0x7f1eb81457f0 2024-03-29T10:44:40.201 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:40.201 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:40.201 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1eb6ffd640 10 monclient: dump: 2024-03-29T10:44:40.201 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:40.201 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:40.201 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:40.201 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:40.201 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:40.201 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:40.202 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:40.202 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:40.202 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:40.202 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:40.202 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1ebe94d640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:40.202 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.215+0000 7f1ebe94d640 4 set_mon_vals no callback set 2024-03-29T10:44:40.202 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.216+0000 7f1ebe94d640 10 monclient: shutdown 2024-03-29T10:44:40.202 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.216+0000 7f1ebe94d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:40.202 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.216+0000 7f1ebe94d640 1 -- 172.21.15.57:0/3500108898 >> v1:172.21.15.57:6789/0 conn(0x7f1eb81457f0 legacy=0x7f1eb813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:40.204 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.217+0000 7f1ebe94d640 1 -- 172.21.15.57:0/3500108898 shutdown_connections 2024-03-29T10:44:40.204 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.217+0000 7f1ebe94d640 1 -- 172.21.15.57:0/3500108898 >> 172.21.15.57:0/3500108898 conn(0x7f1eb8066a00 msgr2=0x7f1eb8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:40.204 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.217+0000 7f1ebe94d640 1 -- 172.21.15.57:0/3500108898 shutdown_connections 2024-03-29T10:44:40.204 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.217+0000 7f1ebe94d640 1 -- 172.21.15.57:0/3500108898 wait complete. 2024-03-29T10:44:40.204 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.218+0000 7f1ebe94d640 10 monclient: build_initial_monmap 2024-03-29T10:44:40.204 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.218+0000 7f1ebe94d640 10 monclient: monmap: 2024-03-29T10:44:40.204 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:44:40.204 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:40.204 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:44:40.204 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:44:40.205 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:40.205 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:40.205 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:44:40.205 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:44:40.205 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:44:40.205 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:40.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.219+0000 7f1ebe94d640 1 librados: starting msgr at 2024-03-29T10:44:40.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.219+0000 7f1ebe94d640 1 librados: starting objecter 2024-03-29T10:44:40.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.219+0000 7f1ebe94d640 1 Processor -- start 2024-03-29T10:44:40.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.219+0000 7f1ebe94d640 1 -- start start 2024-03-29T10:44:40.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.219+0000 7f1ebe94d640 1 librados: setting wanted keys 2024-03-29T10:44:40.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.219+0000 7f1ebe94d640 1 librados: calling monclient init 2024-03-29T10:44:40.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.220+0000 7f1ebe94d640 10 monclient: init 2024-03-29T10:44:40.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.221+0000 7f1ebe94d640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:40.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.221+0000 7f1ebe94d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:44:40.206 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.221+0000 7f1ebe94d640 10 monclient(hunting): picked mon.noname-b con 0x7f1eb80a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T10:44:40.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.224+0000 7f1ebe94d640 10 monclient(hunting): picked mon.noname-a con 0x7f1eb813f650 addr v1:172.21.15.57:6789/0 2024-03-29T10:44:40.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.224+0000 7f1eb7fff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f1eb80a9d70 0x7f1eb8134360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:39805/0 (socket says 172.21.15.57:39805) 2024-03-29T10:44:40.211 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.228+0000 7f1ebe94d640 10 monclient(hunting): picked mon.noname-c con 0x7f1eb81457f0 addr v1:172.21.15.57:6790/0 2024-03-29T10:44:40.211 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.228+0000 7f1ebe94d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb8136550 con 0x7f1eb813f650 2024-03-29T10:44:40.211 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.228+0000 7f1ebe94d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb8131730 con 0x7f1eb80a9d70 2024-03-29T10:44:40.211 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.228+0000 7f1ebe94d640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb8136220 con 0x7f1eb81457f0 2024-03-29T10:44:40.211 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.228+0000 7f1ebe94d640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:40.211 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.228+0000 7f1ebe94d640 10 monclient(hunting): authenticate will time out at 1232.404588s 2024-03-29T10:44:40.211 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.228+0000 7f1eb7fff640 1 -- 172.21.15.57:0/551716169 learned_addr learned my addr 172.21.15.57:0/551716169 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:44:40.211 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.228+0000 7f1eb77fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1eb813f650 0x7f1eb8130910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:50531/0 (socket says 172.21.15.57:50531) 2024-03-29T10:44:40.212 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.230+0000 7f1eb4ff9640 10 client.?.objecter ms_handle_connect 0x7f1eb80a9d70 2024-03-29T10:44:40.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.230+0000 7f1eb4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:40.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.230+0000 7f1eb4ff9640 10 client.?.objecter ms_handle_connect 0x7f1eb81457f0 2024-03-29T10:44:40.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.230+0000 7f1eb4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:40.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.230+0000 7f1eb4ff9640 10 client.?.objecter ms_handle_connect 0x7f1eb813f650 2024-03-29T10:44:40.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.230+0000 7f1eb4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:40.218 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.233+0000 7f1eb4ff9640 1 -- 172.21.15.57:0/551716169 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206481573 0 0) 0x7f1eb8136550 con 0x7f1eb813f650 2024-03-29T10:44:40.218 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.233+0000 7f1eb4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:40.218 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.233+0000 7f1eb4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:40.218 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.233+0000 7f1eb4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:40.218 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.233+0000 7f1eb4ff9640 10 monclient(hunting): my global_id is 4395 2024-03-29T10:44:40.218 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.233+0000 7f1eb4ff9640 1 -- 172.21.15.57:0/551716169 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea4004eb0 con 0x7f1eb813f650 2024-03-29T10:44:40.220 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.235+0000 7f1eb4ff9640 1 -- 172.21.15.57:0/551716169 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2615314575 0 0) 0x7f1eb8131730 con 0x7f1eb80a9d70 2024-03-29T10:44:40.220 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.235+0000 7f1eb4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:40.220 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.235+0000 7f1eb4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:40.220 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.235+0000 7f1eb4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:40.220 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.235+0000 7f1eb4ff9640 10 monclient(hunting): my global_id is 4399 2024-03-29T10:44:40.220 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.235+0000 7f1eb4ff9640 1 -- 172.21.15.57:0/551716169 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1eb8136550 con 0x7f1eb80a9d70 2024-03-29T10:44:40.220 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.235+0000 7f1eb4ff9640 1 -- 172.21.15.57:0/551716169 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 198367126 0 0) 0x7f1ea4004eb0 con 0x7f1eb813f650 2024-03-29T10:44:40.221 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.235+0000 7f1eb4ff9640 1 -- 172.21.15.57:0/551716169 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1eb8131730 con 0x7f1eb813f650 2024-03-29T10:44:40.221 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.235+0000 7f1eb4ff9640 1 -- 172.21.15.57:0/551716169 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1eac003ef0 con 0x7f1eb813f650 2024-03-29T10:44:40.221 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.235+0000 7f1eb4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:40.221 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.235+0000 7f1eb4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:40.221 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.235+0000 7f1eb4ff9640 10 monclient(hunting): dump: 2024-03-29T10:44:40.221 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:40.221 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:40.221 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:40.221 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:40.221 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:40.221 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:40.221 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:40.221 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:40.221 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:40.222 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:40.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.235+0000 7f1eb4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:40.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1eb4ff9640 1 -- 172.21.15.57:0/551716169 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2939574617 0 0) 0x7f1eb8136550 con 0x7f1eb80a9d70 2024-03-29T10:44:40.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1eb4ff9640 1 -- 172.21.15.57:0/551716169 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ea4004eb0 con 0x7f1eb80a9d70 2024-03-29T10:44:40.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1eb4ff9640 1 -- 172.21.15.57:0/551716169 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1ea8001c60 con 0x7f1eb80a9d70 2024-03-29T10:44:40.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1eb4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:40.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1eb4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:40.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1eb4ff9640 10 monclient(hunting): dump: 2024-03-29T10:44:40.222 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:40.222 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:40.222 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:40.222 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:40.222 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:40.223 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:40.223 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:40.223 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:40.223 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:40.223 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:40.223 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1eb4ff9640 1 -- 172.21.15.57:0/551716169 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2511460465 0 0) 0x7f1eb8131730 con 0x7f1eb813f650 2024-03-29T10:44:40.223 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1eb4ff9640 1 -- 172.21.15.57:0/551716169 >> v1:172.21.15.57:6790/0 conn(0x7f1eb81457f0 legacy=0x7f1eb8131020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:40.223 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1eb4ff9640 1 -- 172.21.15.57:0/551716169 >> v1:172.21.15.179:6789/0 conn(0x7f1eb80a9d70 legacy=0x7f1eb8134360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:40.223 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1eb4ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:44:40.223 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1eb4ff9640 1 monclient: found mon.a 2024-03-29T10:44:40.223 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1eb4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:40.223 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1eb4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:44:40.223 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1eb4ff9640 1 -- 172.21.15.57:0/551716169 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb8135b20 con 0x7f1eb813f650 2024-03-29T10:44:40.223 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1eb4ff9640 10 monclient: _finish_auth 0 2024-03-29T10:44:40.223 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1eb4ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:44:40.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1eb4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:40.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1ebe94d640 5 monclient: authenticate success, global_id 4395 2024-03-29T10:44:40.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1ebe94d640 10 monclient: _renew_subs 2024-03-29T10:44:40.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1ebe94d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:44:40.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1ebe94d640 1 -- 172.21.15.57:0/551716169 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb8135e00 con 0x7f1eb813f650 2024-03-29T10:44:40.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1ebe94d640 10 client.4395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:40.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1ebe94d640 10 monclient: _renew_subs 2024-03-29T10:44:40.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1ebe94d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:44:40.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1ebe94d640 1 -- 172.21.15.57:0/551716169 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb8134c50 con 0x7f1eb813f650 2024-03-29T10:44:40.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.236+0000 7f1ebe94d640 1 librados: init done 2024-03-29T10:44:40.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.237+0000 7f1ebe94d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:40.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.237+0000 7f1ebe94d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:40.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.237+0000 7f1ebe94d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:44:40.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.237+0000 7f1ebe94d640 1 -- 172.21.15.57:0/551716169 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb8075f10 con 0x7f1eb813f650 2024-03-29T10:44:40.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.240+0000 7f1eb4ff9640 1 -- 172.21.15.57:0/551716169 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1eac0044c0 con 0x7f1eb813f650 2024-03-29T10:44:40.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.240+0000 7f1eb4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:40.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.240+0000 7f1eb67fc640 4 set_mon_vals no callback set 2024-03-29T10:44:40.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.241+0000 7f1eb4ff9640 1 -- 172.21.15.57:0/551716169 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1eac000ed0 con 0x7f1eb813f650 2024-03-29T10:44:40.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.241+0000 7f1eb4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:40.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.241+0000 7f1eb4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:40.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.241+0000 7f1eb4ff9640 10 monclient: dump: 2024-03-29T10:44:40.227 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:40.227 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:40.227 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:40.227 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:40.227 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:40.227 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:40.227 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:40.227 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:40.227 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:40.227 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:40.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.241+0000 7f1eb4ff9640 1 -- 172.21.15.57:0/551716169 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f1eac0010f0 con 0x7f1eb813f650 2024-03-29T10:44:40.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.242+0000 7f1eb7fff640 10 monclient: get_auth_request con 0x7f1ea406df20 auth_method 0 2024-03-29T10:44:40.228 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.242+0000 7f1eb4ff9640 1 -- 172.21.15.57:0/551716169 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f1eac084730 con 0x7f1eb813f650 2024-03-29T10:44:40.228 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.242+0000 7f1eb4ff9640 10 client.4395.objecter ms_dispatch 0x7f1eb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:40.228 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.242+0000 7f1eb4ff9640 3 client.4395.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:40.228 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.242+0000 7f1eb4ff9640 3 client.4395.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:40.228 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.242+0000 7f1eb4ff9640 20 client.4395.objecter dump_active .. 0 homeless 2024-03-29T10:44:40.228 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.242+0000 7f1eb4ff9640 10 client.4395.objecter ms_handle_connect 0x7f1ea406df20 2024-03-29T10:44:40.235 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.252+0000 7f1eb4ff9640 1 -- 172.21.15.57:0/551716169 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1eac0500d0 con 0x7f1eb813f650 2024-03-29T10:44:40.235 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.252+0000 7f1eb4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:40.235 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.252+0000 7f1eb4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:40.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.583+0000 7f1ebe94d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:40.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.583+0000 7f1ebe94d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:40.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.583+0000 7f1ebe94d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:44:40.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.583+0000 7f1ebe94d640 1 -- 172.21.15.57:0/551716169 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1eb8144a30 con 0x7f1eb813f650 2024-03-29T10:44:40.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.585+0000 7f1eb4ff9640 1 -- 172.21.15.57:0/551716169 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f1eac052000 con 0x7f1eb813f650 2024-03-29T10:44:40.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.585+0000 7f1eb4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:40.569 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.585+0000 7f1eb4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:40.570 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:44:40.570 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:40.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.589+0000 7f1ebe94d640 10 librados: watch_flush enter 2024-03-29T10:44:40.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.589+0000 7f1ebe94d640 10 librados: watch_flush exit 2024-03-29T10:44:40.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.589+0000 7f1ebe94d640 20 client.4395.objecter shutdown clearing up homeless session... 2024-03-29T10:44:40.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.589+0000 7f1ebe94d640 10 client.4395.objecter successfully canceled tick 2024-03-29T10:44:40.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.589+0000 7f1ebe94d640 1 -- 172.21.15.57:0/551716169 >> v1:172.21.15.179:6816/2698295936 conn(0x7f1ea406df20 legacy=0x7f1ea40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:40.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.589+0000 7f1ebe94d640 10 monclient: shutdown 2024-03-29T10:44:40.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.589+0000 7f1ebe94d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:40.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.589+0000 7f1ebe94d640 1 -- 172.21.15.57:0/551716169 >> v1:172.21.15.57:6789/0 conn(0x7f1eb813f650 legacy=0x7f1eb8130910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:40.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.589+0000 7f1ebe94d640 1 -- 172.21.15.57:0/551716169 shutdown_connections 2024-03-29T10:44:40.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.589+0000 7f1ebe94d640 1 -- 172.21.15.57:0/551716169 >> 172.21.15.57:0/551716169 conn(0x7f1eb8066a00 msgr2=0x7f1eb80b60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:40.573 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.589+0000 7f1ebe94d640 1 -- 172.21.15.57:0/551716169 shutdown_connections 2024-03-29T10:44:40.573 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.590+0000 7f1ebe94d640 1 -- 172.21.15.57:0/551716169 wait complete. 2024-03-29T10:44:40.573 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:40.590+0000 7f1ebe94d640 1 librados: shutdown 2024-03-29T10:44:40.592 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:47.398 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:47.556 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.572+0000 7f0b2c24e640 10 monclient: get_monmap_and_config 2024-03-29T10:44:47.556 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.572+0000 7f0b2c24e640 10 monclient: build_initial_monmap 2024-03-29T10:44:47.556 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.572+0000 7f0b2c24e640 10 monclient: monmap: 2024-03-29T10:44:47.556 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:44:47.556 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:47.556 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:44:47.572962+0000 2024-03-29T10:44:47.556 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:44:47.572962+0000 2024-03-29T10:44:47.556 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:47.556 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:47.556 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:44:47.556 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:44:47.557 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:44:47.557 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:47.557 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.575+0000 7f0b2c24e640 1 Processor -- start 2024-03-29T10:44:47.558 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.575+0000 7f0b2c24e640 1 -- start start 2024-03-29T10:44:47.559 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.575+0000 7f0b2c24e640 10 monclient: init 2024-03-29T10:44:47.559 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.576+0000 7f0b2c24e640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:47.559 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.576+0000 7f0b2c24e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:47.559 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.576+0000 7f0b2c24e640 10 monclient(hunting): picked mon.noname-b con 0x7f0b240a9ac0 addr v1:172.21.15.57:6790/0 2024-03-29T10:44:47.559 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.576+0000 7f0b2c24e640 10 monclient(hunting): picked mon.noname-a con 0x7f0b2413ee90 addr v1:172.21.15.57:6789/0 2024-03-29T10:44:47.559 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.576+0000 7f0b2c24e640 10 monclient(hunting): picked mon.noname-c con 0x7f0b240af900 addr v1:172.21.15.179:6789/0 2024-03-29T10:44:47.559 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.576+0000 7f0b2c24e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b240afd10 con 0x7f0b2413ee90 2024-03-29T10:44:47.559 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.576+0000 7f0b2c24e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b240af6f0 con 0x7f0b240af900 2024-03-29T10:44:47.559 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.576+0000 7f0b2c24e640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b240aac80 con 0x7f0b240a9ac0 2024-03-29T10:44:47.559 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.576+0000 7f0b2c24e640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:47.559 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.576+0000 7f0b2c24e640 10 monclient(hunting): authenticate will time out at 1239.752261s 2024-03-29T10:44:47.562 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.580+0000 7f0b297e9640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f0b2413ee90 0x7f0b2413b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:56719/0 (socket says 172.21.15.57:56719) 2024-03-29T10:44:47.563 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.580+0000 7f0b297e9640 1 -- 172.21.15.57:0/2667632961 learned_addr learned my addr 172.21.15.57:0/2667632961 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:44:47.567 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.583+0000 7f0b28fe8640 1 -- 172.21.15.57:0/2667632961 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 273766126 0 0) 0x7f0b240aac80 con 0x7f0b240a9ac0 2024-03-29T10:44:47.567 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.583+0000 7f0b28fe8640 10 monclient(hunting): _negotiate 2024-03-29T10:44:47.567 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.583+0000 7f0b28fe8640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:47.567 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.583+0000 7f0b28fe8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:47.567 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.583+0000 7f0b28fe8640 10 monclient(hunting): my global_id is 4424 2024-03-29T10:44:47.567 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.583+0000 7f0b28fe8640 1 -- 172.21.15.57:0/2667632961 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b10003520 con 0x7f0b240a9ac0 2024-03-29T10:44:47.567 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.583+0000 7f0b28fe8640 1 -- 172.21.15.57:0/2667632961 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166175398 0 0) 0x7f0b240afd10 con 0x7f0b2413ee90 2024-03-29T10:44:47.567 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.583+0000 7f0b28fe8640 10 monclient(hunting): _negotiate 2024-03-29T10:44:47.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.583+0000 7f0b28fe8640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:47.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.583+0000 7f0b28fe8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:47.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.583+0000 7f0b28fe8640 10 monclient(hunting): my global_id is 4398 2024-03-29T10:44:47.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.583+0000 7f0b28fe8640 1 -- 172.21.15.57:0/2667632961 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b240aac80 con 0x7f0b2413ee90 2024-03-29T10:44:47.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.584+0000 7f0b28fe8640 1 -- 172.21.15.57:0/2667632961 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 462628932 0 0) 0x7f0b10003520 con 0x7f0b240a9ac0 2024-03-29T10:44:47.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.584+0000 7f0b28fe8640 1 -- 172.21.15.57:0/2667632961 >> v1:172.21.15.179:6789/0 conn(0x7f0b240af900 legacy=0x7f0b24149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:44:47.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.584+0000 7f0b28fe8640 1 -- 172.21.15.57:0/2667632961 >> v1:172.21.15.57:6789/0 conn(0x7f0b2413ee90 legacy=0x7f0b2413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:47.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.584+0000 7f0b28fe8640 10 monclient: _finish_hunting 0 2024-03-29T10:44:47.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.584+0000 7f0b28fe8640 1 monclient: found mon.noname-b 2024-03-29T10:44:47.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.584+0000 7f0b28fe8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:47.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.584+0000 7f0b28fe8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:44:47.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.584+0000 7f0b28fe8640 1 -- 172.21.15.57:0/2667632961 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b240aaf60 con 0x7f0b240a9ac0 2024-03-29T10:44:47.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.584+0000 7f0b28fe8640 10 monclient: _finish_auth 0 2024-03-29T10:44:47.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.584+0000 7f0b28fe8640 10 monclient: _check_auth_tickets 2024-03-29T10:44:47.568 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.584+0000 7f0b28fe8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:47.569 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.584+0000 7f0b2c24e640 5 monclient: authenticate success, global_id 4424 2024-03-29T10:44:47.569 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.584+0000 7f0b2c24e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:47.569 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.584+0000 7f0b28fe8640 1 -- 172.21.15.57:0/2667632961 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f0b20001690 con 0x7f0b240a9ac0 2024-03-29T10:44:47.569 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.584+0000 7f0b28fe8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:47.569 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.584+0000 7f0b28fe8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:47.569 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.584+0000 7f0b28fe8640 10 monclient: dump: 2024-03-29T10:44:47.569 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:47.569 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:47.569 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:47.569 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:47.569 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:47.569 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:47.569 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:47.569 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:47.569 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:47.569 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:47.570 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.584+0000 7f0b2c24e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:47.571 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.584+0000 7f0b28fe8640 1 -- 172.21.15.57:0/2667632961 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b200017f0 con 0x7f0b240a9ac0 2024-03-29T10:44:47.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.584+0000 7f0b28fe8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:47.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.584+0000 7f0b2c24e640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:47.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.584+0000 7f0b2c24e640 4 set_mon_vals no callback set 2024-03-29T10:44:47.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.585+0000 7f0b2c24e640 10 monclient: shutdown 2024-03-29T10:44:47.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.585+0000 7f0b2c24e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:47.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.585+0000 7f0b2c24e640 1 -- 172.21.15.57:0/2667632961 >> v1:172.21.15.57:6790/0 conn(0x7f0b240a9ac0 legacy=0x7f0b240adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:47.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.585+0000 7f0b2c24e640 1 -- 172.21.15.57:0/2667632961 shutdown_connections 2024-03-29T10:44:47.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.585+0000 7f0b2c24e640 1 -- 172.21.15.57:0/2667632961 >> 172.21.15.57:0/2667632961 conn(0x7f0b240b35e0 msgr2=0x7f0b240b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:47.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.586+0000 7f0b2c24e640 1 -- 172.21.15.57:0/2667632961 shutdown_connections 2024-03-29T10:44:47.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.586+0000 7f0b2c24e640 1 -- 172.21.15.57:0/2667632961 wait complete. 2024-03-29T10:44:47.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.587+0000 7f0b2c24e640 10 monclient: build_initial_monmap 2024-03-29T10:44:47.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.587+0000 7f0b2c24e640 10 monclient: monmap: 2024-03-29T10:44:47.572 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:44:47.572 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:47.573 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:44:47.573 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:44:47.573 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:47.573 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:47.573 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:44:47.573 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:44:47.573 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:44:47.573 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:47.573 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.587+0000 7f0b2c24e640 1 librados: starting msgr at 2024-03-29T10:44:47.573 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.587+0000 7f0b2c24e640 1 librados: starting objecter 2024-03-29T10:44:47.573 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.588+0000 7f0b2c24e640 1 Processor -- start 2024-03-29T10:44:47.573 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.588+0000 7f0b2c24e640 1 -- start start 2024-03-29T10:44:47.573 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.588+0000 7f0b2c24e640 1 librados: setting wanted keys 2024-03-29T10:44:47.573 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.588+0000 7f0b2c24e640 1 librados: calling monclient init 2024-03-29T10:44:47.574 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.588+0000 7f0b2c24e640 10 monclient: init 2024-03-29T10:44:47.574 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.588+0000 7f0b2c24e640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:47.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.588+0000 7f0b2c24e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:47.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.588+0000 7f0b2c24e640 10 monclient(hunting): picked mon.noname-a con 0x7f0b240af900 addr v1:172.21.15.57:6789/0 2024-03-29T10:44:47.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.588+0000 7f0b2c24e640 10 monclient(hunting): picked mon.noname-b con 0x7f0b2413ee90 addr v1:172.21.15.179:6789/0 2024-03-29T10:44:47.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.588+0000 7f0b2c24e640 10 monclient(hunting): picked mon.noname-c con 0x7f0b24140a60 addr v1:172.21.15.57:6790/0 2024-03-29T10:44:47.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.588+0000 7f0b2c24e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b24141ef0 con 0x7f0b240af900 2024-03-29T10:44:47.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.588+0000 7f0b2c24e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b241421d0 con 0x7f0b2413ee90 2024-03-29T10:44:47.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.588+0000 7f0b2c24e640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b241424b0 con 0x7f0b24140a60 2024-03-29T10:44:47.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.588+0000 7f0b2c24e640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:47.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.588+0000 7f0b2c24e640 10 monclient(hunting): authenticate will time out at 1239.764980s 2024-03-29T10:44:47.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.595+0000 7f0b2a7eb640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f0b24140a60 0x7f0b241e70e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:45367/0 (socket says 172.21.15.57:45367) 2024-03-29T10:44:47.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.595+0000 7f0b2a7eb640 1 -- 172.21.15.57:0/1037846349 learned_addr learned my addr 172.21.15.57:0/1037846349 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:44:47.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.595+0000 7f0b1a7fc640 10 client.?.objecter ms_handle_connect 0x7f0b24140a60 2024-03-29T10:44:47.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.595+0000 7f0b1a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:47.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.595+0000 7f0b1a7fc640 1 -- 172.21.15.57:0/1037846349 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440746401 0 0) 0x7f0b241424b0 con 0x7f0b24140a60 2024-03-29T10:44:47.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.595+0000 7f0b1a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:47.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.595+0000 7f0b1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:47.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.595+0000 7f0b1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:47.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.595+0000 7f0b1a7fc640 10 monclient(hunting): my global_id is 4427 2024-03-29T10:44:47.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.595+0000 7f0b1a7fc640 1 -- 172.21.15.57:0/1037846349 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b08004eb0 con 0x7f0b24140a60 2024-03-29T10:44:47.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.595+0000 7f0b1a7fc640 1 -- 172.21.15.57:0/1037846349 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2317279992 0 0) 0x7f0b08004eb0 con 0x7f0b24140a60 2024-03-29T10:44:47.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.595+0000 7f0b1a7fc640 1 -- 172.21.15.57:0/1037846349 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b241424b0 con 0x7f0b24140a60 2024-03-29T10:44:47.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.595+0000 7f0b1a7fc640 1 -- 172.21.15.57:0/1037846349 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f0b1c004380 con 0x7f0b24140a60 2024-03-29T10:44:47.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.595+0000 7f0b1a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:47.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.595+0000 7f0b1a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:47.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.595+0000 7f0b1a7fc640 10 monclient(hunting): dump: 2024-03-29T10:44:47.585 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:47.585 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:47.585 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:47.585 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:47.585 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:47.585 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:47.585 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:47.585 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:47.586 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:47.586 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:47.586 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.595+0000 7f0b1a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:47.586 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b1a7fc640 1 -- 172.21.15.57:0/1037846349 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3900503634 0 0) 0x7f0b241424b0 con 0x7f0b24140a60 2024-03-29T10:44:47.586 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b1a7fc640 1 -- 172.21.15.57:0/1037846349 >> v1:172.21.15.179:6789/0 conn(0x7f0b2413ee90 legacy=0x7f0b241361e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:44:47.586 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b1a7fc640 1 -- 172.21.15.57:0/1037846349 >> v1:172.21.15.57:6789/0 conn(0x7f0b240af900 legacy=0x7f0b2413c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:47.586 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b1a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:44:47.586 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b1a7fc640 1 monclient: found mon.c 2024-03-29T10:44:47.586 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:47.586 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b1a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:44:47.586 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b1a7fc640 1 -- 172.21.15.57:0/1037846349 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b24140ef0 con 0x7f0b24140a60 2024-03-29T10:44:47.586 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b1a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:44:47.586 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b1a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:44:47.586 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:47.587 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b1a7fc640 1 -- 172.21.15.57:0/1037846349 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b1c0044e0 con 0x7f0b24140a60 2024-03-29T10:44:47.587 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:47.587 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b1a7fc640 1 -- 172.21.15.57:0/1037846349 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f0b1c0014f0 con 0x7f0b24140a60 2024-03-29T10:44:47.587 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:47.587 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b1a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:47.587 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b1a7fc640 10 monclient: dump: 2024-03-29T10:44:47.587 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:47.587 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:47.587 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:47.587 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:47.587 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:47.587 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:47.587 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:47.587 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:47.587 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:47.588 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:47.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b1bfff640 4 set_mon_vals no callback set 2024-03-29T10:44:47.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b2c24e640 5 monclient: authenticate success, global_id 4427 2024-03-29T10:44:47.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b2c24e640 10 monclient: _renew_subs 2024-03-29T10:44:47.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b2c24e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:44:47.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b2c24e640 1 -- 172.21.15.57:0/1037846349 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b24141410 con 0x7f0b24140a60 2024-03-29T10:44:47.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b2c24e640 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:47.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b2c24e640 10 monclient: _renew_subs 2024-03-29T10:44:47.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b2c24e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:44:47.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b2c24e640 1 -- 172.21.15.57:0/1037846349 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2413d1e0 con 0x7f0b24140a60 2024-03-29T10:44:47.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.596+0000 7f0b2c24e640 1 librados: init done 2024-03-29T10:44:47.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.598+0000 7f0b2c24e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:47.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.598+0000 7f0b2c24e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:47.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.598+0000 7f0b2c24e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:44:47.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.598+0000 7f0b2c24e640 1 -- 172.21.15.57:0/1037846349 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b24141600 con 0x7f0b24140a60 2024-03-29T10:44:47.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.599+0000 7f0b1a7fc640 1 -- 172.21.15.57:0/1037846349 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f0b1c001650 con 0x7f0b24140a60 2024-03-29T10:44:47.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.599+0000 7f0b1a7fc640 1 -- 172.21.15.57:0/1037846349 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f0b1c084e00 con 0x7f0b24140a60 2024-03-29T10:44:47.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.599+0000 7f0b1a7fc640 10 client.4427.objecter ms_dispatch 0x7f0b24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:47.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.599+0000 7f0b1a7fc640 3 client.4427.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:47.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.599+0000 7f0b1a7fc640 3 client.4427.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:47.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.599+0000 7f0b1a7fc640 20 client.4427.objecter dump_active .. 0 homeless 2024-03-29T10:44:47.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.599+0000 7f0b29fea640 10 monclient: get_auth_request con 0x7f0b0806dc90 auth_method 0 2024-03-29T10:44:47.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.600+0000 7f0b1a7fc640 10 client.4427.objecter ms_handle_connect 0x7f0b0806dc90 2024-03-29T10:44:47.591 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.607+0000 7f0b1a7fc640 1 -- 172.21.15.57:0/1037846349 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b1c050510 con 0x7f0b24140a60 2024-03-29T10:44:47.591 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.607+0000 7f0b1a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:47.591 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.607+0000 7f0b1a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:47.908 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.925+0000 7f0b2c24e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:47.908 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.925+0000 7f0b2c24e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:47.908 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.925+0000 7f0b2c24e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:44:47.908 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.925+0000 7f0b2c24e640 1 -- 172.21.15.57:0/1037846349 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b24075f10 con 0x7f0b24140a60 2024-03-29T10:44:47.912 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.929+0000 7f0b1a7fc640 1 -- 172.21.15.57:0/1037846349 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f0b1c052610 con 0x7f0b24140a60 2024-03-29T10:44:47.912 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.929+0000 7f0b1a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:47.912 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.929+0000 7f0b1a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:47.915 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:44:47.915 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:47.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.936+0000 7f0b2c24e640 10 librados: watch_flush enter 2024-03-29T10:44:47.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.936+0000 7f0b2c24e640 10 librados: watch_flush exit 2024-03-29T10:44:47.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.936+0000 7f0b2c24e640 20 client.4427.objecter shutdown clearing up homeless session... 2024-03-29T10:44:47.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.936+0000 7f0b2c24e640 10 client.4427.objecter successfully canceled tick 2024-03-29T10:44:47.920 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.936+0000 7f0b2c24e640 1 -- 172.21.15.57:0/1037846349 >> v1:172.21.15.179:6816/2698295936 conn(0x7f0b0806dc90 legacy=0x7f0b08070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:47.920 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.936+0000 7f0b2c24e640 10 monclient: shutdown 2024-03-29T10:44:47.920 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.936+0000 7f0b2c24e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:47.920 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.936+0000 7f0b2c24e640 1 -- 172.21.15.57:0/1037846349 >> v1:172.21.15.57:6790/0 conn(0x7f0b24140a60 legacy=0x7f0b241e70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:47.920 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.936+0000 7f0b2c24e640 1 -- 172.21.15.57:0/1037846349 shutdown_connections 2024-03-29T10:44:47.920 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.936+0000 7f0b2c24e640 1 -- 172.21.15.57:0/1037846349 >> 172.21.15.57:0/1037846349 conn(0x7f0b240b35e0 msgr2=0x7f0b240adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:47.921 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.936+0000 7f0b2c24e640 1 -- 172.21.15.57:0/1037846349 shutdown_connections 2024-03-29T10:44:47.921 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.936+0000 7f0b2c24e640 1 -- 172.21.15.57:0/1037846349 wait complete. 2024-03-29T10:44:47.921 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:47.937+0000 7f0b2c24e640 1 librados: shutdown 2024-03-29T10:44:47.938 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:54.745 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:54.926 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.943+0000 7fc234a74640 10 monclient: get_monmap_and_config 2024-03-29T10:44:54.927 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.943+0000 7fc234a74640 10 monclient: build_initial_monmap 2024-03-29T10:44:54.927 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.943+0000 7fc234a74640 10 monclient: monmap: 2024-03-29T10:44:54.927 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:44:54.927 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:54.927 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:44:54.944450+0000 2024-03-29T10:44:54.927 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:44:54.944450+0000 2024-03-29T10:44:54.927 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:54.927 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:54.927 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:44:54.927 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:44:54.927 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:44:54.927 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:54.933 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.945+0000 7fc234a74640 1 Processor -- start 2024-03-29T10:44:54.933 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.945+0000 7fc234a74640 1 -- start start 2024-03-29T10:44:54.933 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.945+0000 7fc234a74640 10 monclient: init 2024-03-29T10:44:54.933 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.946+0000 7fc234a74640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:54.933 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.946+0000 7fc234a74640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:44:54.933 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.946+0000 7fc234a74640 10 monclient(hunting): picked mon.noname-a con 0x7fc2300a9750 addr v1:172.21.15.57:6789/0 2024-03-29T10:44:54.933 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.946+0000 7fc234a74640 10 monclient(hunting): picked mon.noname-b con 0x7fc2300aad20 addr v1:172.21.15.57:6790/0 2024-03-29T10:44:54.933 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.946+0000 7fc234a74640 10 monclient(hunting): picked mon.noname-c con 0x7fc2300a9b30 addr v1:172.21.15.179:6789/0 2024-03-29T10:44:54.933 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.946+0000 7fc234a74640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2300af4d0 con 0x7fc2300a9750 2024-03-29T10:44:54.933 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.946+0000 7fc234a74640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2300af7b0 con 0x7fc2300a9b30 2024-03-29T10:44:54.934 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.946+0000 7fc234a74640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2300acab0 con 0x7fc2300aad20 2024-03-29T10:44:54.934 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.946+0000 7fc234a74640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:54.934 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.946+0000 7fc234a74640 10 monclient(hunting): authenticate will time out at 1247.122156s 2024-03-29T10:44:54.934 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.949+0000 7fc22ffff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fc2300a9b30 0x7fc230149930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:56183/0 (socket says 172.21.15.57:56183) 2024-03-29T10:44:54.934 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.949+0000 7fc22ffff640 1 -- 172.21.15.57:0/2081708057 learned_addr learned my addr 172.21.15.57:0/2081708057 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:44:54.936 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.951+0000 7fc22e7fc640 1 -- 172.21.15.57:0/2081708057 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2869661990 0 0) 0x7fc2300af7b0 con 0x7fc2300a9b30 2024-03-29T10:44:54.936 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.951+0000 7fc22e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:54.936 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.951+0000 7fc22e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:54.936 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.951+0000 7fc22e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:54.936 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.951+0000 7fc22e7fc640 10 monclient(hunting): my global_id is 4402 2024-03-29T10:44:54.936 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.951+0000 7fc22e7fc640 1 -- 172.21.15.57:0/2081708057 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc218003520 con 0x7fc2300a9b30 2024-03-29T10:44:54.937 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.954+0000 7fc22e7fc640 1 -- 172.21.15.57:0/2081708057 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 504907357 0 0) 0x7fc218003520 con 0x7fc2300a9b30 2024-03-29T10:44:54.937 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.954+0000 7fc22e7fc640 1 -- 172.21.15.57:0/2081708057 >> v1:172.21.15.57:6790/0 conn(0x7fc2300aad20 legacy=0x7fc230141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:44:54.937 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.954+0000 7fc22e7fc640 1 -- 172.21.15.57:0/2081708057 >> v1:172.21.15.57:6789/0 conn(0x7fc2300a9750 legacy=0x7fc2300ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:44:54.937 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.954+0000 7fc22e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:44:54.937 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.954+0000 7fc22e7fc640 1 monclient: found mon.noname-c 2024-03-29T10:44:54.937 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.954+0000 7fc22e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:54.937 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.954+0000 7fc22e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:44:54.937 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.954+0000 7fc22e7fc640 1 -- 172.21.15.57:0/2081708057 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2300aba30 con 0x7fc2300a9b30 2024-03-29T10:44:54.937 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.954+0000 7fc22e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:44:54.937 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.954+0000 7fc22e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:44:54.937 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.954+0000 7fc22e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:54.937 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.954+0000 7fc234a74640 5 monclient: authenticate success, global_id 4402 2024-03-29T10:44:54.937 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.954+0000 7fc22e7fc640 1 -- 172.21.15.57:0/2081708057 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fc224001690 con 0x7fc2300a9b30 2024-03-29T10:44:54.937 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.954+0000 7fc22e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:54.938 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.954+0000 7fc22e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:54.938 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.954+0000 7fc22e7fc640 10 monclient: dump: 2024-03-29T10:44:54.938 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:54.938 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:54.938 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:54.938 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:54.938 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:54.938 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:54.938 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:54.938 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:54.938 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:54.938 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:54.938 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.954+0000 7fc234a74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:54.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.954+0000 7fc22e7fc640 1 -- 172.21.15.57:0/2081708057 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2240017f0 con 0x7fc2300a9b30 2024-03-29T10:44:54.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.954+0000 7fc22e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:54.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.955+0000 7fc234a74640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:54.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.955+0000 7fc234a74640 4 set_mon_vals no callback set 2024-03-29T10:44:54.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.955+0000 7fc234a74640 10 monclient: shutdown 2024-03-29T10:44:54.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.955+0000 7fc234a74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:54.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.955+0000 7fc234a74640 1 -- 172.21.15.57:0/2081708057 >> v1:172.21.15.179:6789/0 conn(0x7fc2300a9b30 legacy=0x7fc230149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:54.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.956+0000 7fc234a74640 1 -- 172.21.15.57:0/2081708057 shutdown_connections 2024-03-29T10:44:54.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.956+0000 7fc234a74640 1 -- 172.21.15.57:0/2081708057 >> 172.21.15.57:0/2081708057 conn(0x7fc2300673e0 msgr2=0x7fc2300b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:54.941 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.959+0000 7fc234a74640 1 -- 172.21.15.57:0/2081708057 shutdown_connections 2024-03-29T10:44:54.944 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.962+0000 7fc234a74640 1 -- 172.21.15.57:0/2081708057 wait complete. 2024-03-29T10:44:54.952 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.963+0000 7fc234a74640 10 monclient: build_initial_monmap 2024-03-29T10:44:54.952 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.963+0000 7fc234a74640 10 monclient: monmap: 2024-03-29T10:44:54.952 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:44:54.952 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:54.952 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:44:54.952 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:44:54.952 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:54.952 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:54.953 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:44:54.953 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:44:54.953 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:44:54.953 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:54.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.963+0000 7fc234a74640 1 librados: starting msgr at 2024-03-29T10:44:54.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.963+0000 7fc234a74640 1 librados: starting objecter 2024-03-29T10:44:54.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.964+0000 7fc234a74640 1 Processor -- start 2024-03-29T10:44:54.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.964+0000 7fc234a74640 1 -- start start 2024-03-29T10:44:54.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.964+0000 7fc234a74640 1 librados: setting wanted keys 2024-03-29T10:44:54.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.964+0000 7fc234a74640 1 librados: calling monclient init 2024-03-29T10:44:54.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.964+0000 7fc234a74640 10 monclient: init 2024-03-29T10:44:54.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.965+0000 7fc234a74640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:54.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.965+0000 7fc234a74640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:54.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.965+0000 7fc234a74640 10 monclient(hunting): picked mon.noname-a con 0x7fc2300a9b30 addr v1:172.21.15.57:6789/0 2024-03-29T10:44:54.954 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.965+0000 7fc234a74640 10 monclient(hunting): picked mon.noname-b con 0x7fc2301c8370 addr v1:172.21.15.179:6789/0 2024-03-29T10:44:54.954 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.965+0000 7fc234a74640 10 monclient(hunting): picked mon.noname-c con 0x7fc2301c90d0 addr v1:172.21.15.57:6790/0 2024-03-29T10:44:54.954 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.965+0000 7fc234a74640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2301c8800 con 0x7fc2300a9b30 2024-03-29T10:44:54.954 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.965+0000 7fc234a74640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2301f8c70 con 0x7fc2301c8370 2024-03-29T10:44:54.954 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.965+0000 7fc234a74640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2301c79e0 con 0x7fc2301c90d0 2024-03-29T10:44:54.954 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.965+0000 7fc234a74640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:54.954 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.965+0000 7fc234a74640 10 monclient(hunting): authenticate will time out at 1247.141387s 2024-03-29T10:44:54.954 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.971+0000 7fc22effd640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fc2301c8370 0x7fc2301c89c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:56303/0 (socket says 172.21.15.57:56303) 2024-03-29T10:44:54.955 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.971+0000 7fc22effd640 1 -- 172.21.15.57:0/3322172428 learned_addr learned my addr 172.21.15.57:0/3322172428 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:44:54.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.973+0000 7fc217fff640 10 client.?.objecter ms_handle_connect 0x7fc2300a9b30 2024-03-29T10:44:54.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.973+0000 7fc217fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:54.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.973+0000 7fc217fff640 10 client.?.objecter ms_handle_connect 0x7fc2301c8370 2024-03-29T10:44:54.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.973+0000 7fc217fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:54.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.973+0000 7fc217fff640 1 -- 172.21.15.57:0/3322172428 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766700745 0 0) 0x7fc220001090 con 0x7fc2300a9b30 2024-03-29T10:44:54.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.973+0000 7fc217fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:54.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.973+0000 7fc217fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:54.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.973+0000 7fc217fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:54.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.973+0000 7fc217fff640 10 monclient(hunting): my global_id is 4401 2024-03-29T10:44:54.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.973+0000 7fc217fff640 1 -- 172.21.15.57:0/3322172428 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc21c004eb0 con 0x7fc2300a9b30 2024-03-29T10:44:54.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.973+0000 7fc217fff640 10 client.?.objecter ms_handle_connect 0x7fc2301c90d0 2024-03-29T10:44:54.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.973+0000 7fc217fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:54.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.974+0000 7fc217fff640 1 -- 172.21.15.57:0/3322172428 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 745745983 0 0) 0x7fc21c004eb0 con 0x7fc2300a9b30 2024-03-29T10:44:54.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.974+0000 7fc217fff640 1 -- 172.21.15.57:0/3322172428 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc220001090 con 0x7fc2300a9b30 2024-03-29T10:44:54.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.977+0000 7fc217fff640 1 -- 172.21.15.57:0/3322172428 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522240813 0 0) 0x7fc2301c79e0 con 0x7fc2301c90d0 2024-03-29T10:44:54.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.977+0000 7fc217fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:54.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.977+0000 7fc217fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:54.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.977+0000 7fc217fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:54.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.977+0000 7fc217fff640 10 monclient(hunting): my global_id is 4430 2024-03-29T10:44:54.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.977+0000 7fc217fff640 1 -- 172.21.15.57:0/3322172428 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc21c004eb0 con 0x7fc2301c90d0 2024-03-29T10:44:54.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.977+0000 7fc217fff640 1 -- 172.21.15.57:0/3322172428 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fc220001860 con 0x7fc2300a9b30 2024-03-29T10:44:54.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.977+0000 7fc217fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:54.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.977+0000 7fc217fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:54.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.977+0000 7fc217fff640 10 monclient(hunting): dump: 2024-03-29T10:44:54.961 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:54.961 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:54.961 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:54.961 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:54.961 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:54.961 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:54.961 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:54.961 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:54.961 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:54.961 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:54.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.977+0000 7fc217fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:54.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.977+0000 7fc217fff640 1 -- 172.21.15.57:0/3322172428 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544180101 0 0) 0x7fc2301f8c70 con 0x7fc2301c8370 2024-03-29T10:44:54.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.977+0000 7fc217fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:54.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.977+0000 7fc217fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:54.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.977+0000 7fc217fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:54.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.977+0000 7fc217fff640 10 monclient(hunting): my global_id is 4405 2024-03-29T10:44:54.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.977+0000 7fc217fff640 1 -- 172.21.15.57:0/3322172428 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2301c79e0 con 0x7fc2301c8370 2024-03-29T10:44:54.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.978+0000 7fc217fff640 1 -- 172.21.15.57:0/3322172428 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3282723224 0 0) 0x7fc2301c79e0 con 0x7fc2301c8370 2024-03-29T10:44:54.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.978+0000 7fc217fff640 1 -- 172.21.15.57:0/3322172428 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2301f8c70 con 0x7fc2301c8370 2024-03-29T10:44:54.963 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.978+0000 7fc217fff640 1 -- 172.21.15.57:0/3322172428 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fc228001af0 con 0x7fc2301c8370 2024-03-29T10:44:54.963 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.978+0000 7fc217fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:54.963 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.978+0000 7fc217fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:54.963 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.978+0000 7fc217fff640 10 monclient(hunting): dump: 2024-03-29T10:44:54.963 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:54.963 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:54.963 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:54.963 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:54.963 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:54.963 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:54.963 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:54.963 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:54.963 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:54.963 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:54.963 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.979+0000 7fc217fff640 1 -- 172.21.15.57:0/3322172428 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3872156399 0 0) 0x7fc2301f8c70 con 0x7fc2301c8370 2024-03-29T10:44:54.964 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.979+0000 7fc217fff640 1 -- 172.21.15.57:0/3322172428 >> v1:172.21.15.57:6790/0 conn(0x7fc2301c90d0 legacy=0x7fc2301ff620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:54.964 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.979+0000 7fc217fff640 1 -- 172.21.15.57:0/3322172428 >> v1:172.21.15.57:6789/0 conn(0x7fc2300a9b30 legacy=0x7fc2301f8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:54.964 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.979+0000 7fc217fff640 10 monclient: _finish_hunting 0 2024-03-29T10:44:54.964 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.979+0000 7fc217fff640 1 monclient: found mon.b 2024-03-29T10:44:54.964 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.979+0000 7fc217fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:54.964 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.979+0000 7fc217fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:44:54.964 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.979+0000 7fc217fff640 1 -- 172.21.15.57:0/3322172428 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2301c9530 con 0x7fc2301c8370 2024-03-29T10:44:54.964 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.979+0000 7fc217fff640 10 monclient: _finish_auth 0 2024-03-29T10:44:54.964 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.979+0000 7fc217fff640 10 monclient: _check_auth_tickets 2024-03-29T10:44:54.964 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.979+0000 7fc217fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:54.965 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.979+0000 7fc234a74640 5 monclient: authenticate success, global_id 4405 2024-03-29T10:44:54.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.979+0000 7fc234a74640 10 monclient: _renew_subs 2024-03-29T10:44:54.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.979+0000 7fc234a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:44:54.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.979+0000 7fc234a74640 1 -- 172.21.15.57:0/3322172428 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc230200d40 con 0x7fc2301c8370 2024-03-29T10:44:54.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.979+0000 7fc234a74640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:54.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.979+0000 7fc234a74640 10 monclient: _renew_subs 2024-03-29T10:44:54.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.979+0000 7fc234a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:44:54.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.979+0000 7fc234a74640 1 -- 172.21.15.57:0/3322172428 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2302010b0 con 0x7fc2301c8370 2024-03-29T10:44:54.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.979+0000 7fc234a74640 1 librados: init done 2024-03-29T10:44:54.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.980+0000 7fc217fff640 1 -- 172.21.15.57:0/3322172428 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc228002000 con 0x7fc2301c8370 2024-03-29T10:44:54.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.980+0000 7fc217fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:54.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.980+0000 7fc217fff640 1 -- 172.21.15.57:0/3322172428 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fc228003b10 con 0x7fc2301c8370 2024-03-29T10:44:54.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.980+0000 7fc217fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:54.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.980+0000 7fc217fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:54.967 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.980+0000 7fc217fff640 10 monclient: dump: 2024-03-29T10:44:54.967 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:44:54.967 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:44:54.967 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:54.967 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:44:54.967 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:54.967 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:44:54.967 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:44:54.967 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:44:54.967 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:44:54.967 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:44:54.967 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.980+0000 7fc22dffb640 4 set_mon_vals no callback set 2024-03-29T10:44:54.967 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.980+0000 7fc234a74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:54.967 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.981+0000 7fc234a74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:54.968 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.981+0000 7fc234a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:44:54.968 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.981+0000 7fc234a74640 1 -- 172.21.15.57:0/3322172428 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2301f8ac0 con 0x7fc2301c8370 2024-03-29T10:44:54.968 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.983+0000 7fc217fff640 1 -- 172.21.15.57:0/3322172428 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fc228003c70 con 0x7fc2301c8370 2024-03-29T10:44:54.968 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.984+0000 7fc22f7fe640 10 monclient: get_auth_request con 0x7fc21c06dfd0 auth_method 0 2024-03-29T10:44:54.968 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.984+0000 7fc217fff640 1 -- 172.21.15.57:0/3322172428 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7fc228084be0 con 0x7fc2301c8370 2024-03-29T10:44:54.968 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.984+0000 7fc217fff640 10 client.4405.objecter ms_dispatch 0x7fc230000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:54.968 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.984+0000 7fc217fff640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:54.968 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.984+0000 7fc217fff640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:54.968 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.984+0000 7fc217fff640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-29T10:44:54.968 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.984+0000 7fc217fff640 10 client.4405.objecter ms_handle_connect 0x7fc21c06dfd0 2024-03-29T10:44:54.971 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.989+0000 7fc217fff640 1 -- 172.21.15.57:0/3322172428 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc2280502f0 con 0x7fc2301c8370 2024-03-29T10:44:54.971 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.989+0000 7fc217fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:54.971 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:54.989+0000 7fc217fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:55.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:55.355+0000 7fc234a74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:55.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:55.355+0000 7fc234a74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:55.338 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:55.355+0000 7fc234a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:44:55.338 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:55.355+0000 7fc234a74640 1 -- 172.21.15.57:0/3322172428 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc23004f730 con 0x7fc2301c8370 2024-03-29T10:44:55.338 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:55.356+0000 7fc217fff640 1 -- 172.21.15.57:0/3322172428 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7fc2280523f0 con 0x7fc2301c8370 2024-03-29T10:44:55.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:55.356+0000 7fc217fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:55.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:55.356+0000 7fc217fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:55.339 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:44:55.339 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:55.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:55.359+0000 7fc234a74640 10 librados: watch_flush enter 2024-03-29T10:44:55.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:55.360+0000 7fc234a74640 10 librados: watch_flush exit 2024-03-29T10:44:55.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:55.360+0000 7fc234a74640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-29T10:44:55.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:55.360+0000 7fc234a74640 10 client.4405.objecter successfully canceled tick 2024-03-29T10:44:55.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:55.360+0000 7fc234a74640 1 -- 172.21.15.57:0/3322172428 >> v1:172.21.15.179:6816/2698295936 conn(0x7fc21c06dfd0 legacy=0x7fc21c070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:55.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:55.360+0000 7fc234a74640 10 monclient: shutdown 2024-03-29T10:44:55.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:55.360+0000 7fc234a74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:55.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:55.360+0000 7fc234a74640 1 -- 172.21.15.57:0/3322172428 >> v1:172.21.15.179:6789/0 conn(0x7fc2301c8370 legacy=0x7fc2301c89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:55.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:55.360+0000 7fc234a74640 1 -- 172.21.15.57:0/3322172428 shutdown_connections 2024-03-29T10:44:55.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:55.360+0000 7fc234a74640 1 -- 172.21.15.57:0/3322172428 >> 172.21.15.57:0/3322172428 conn(0x7fc2300673e0 msgr2=0x7fc230069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:55.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:55.360+0000 7fc234a74640 1 -- 172.21.15.57:0/3322172428 shutdown_connections 2024-03-29T10:44:55.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:55.360+0000 7fc234a74640 1 -- 172.21.15.57:0/3322172428 wait complete. 2024-03-29T10:44:55.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:44:55.360+0000 7fc234a74640 1 librados: shutdown 2024-03-29T10:44:55.357 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:02.164 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:02.309 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.325+0000 7f41f9624640 10 monclient: get_monmap_and_config 2024-03-29T10:45:02.309 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.325+0000 7f41f9624640 10 monclient: build_initial_monmap 2024-03-29T10:45:02.309 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.325+0000 7f41f9624640 10 monclient: monmap: 2024-03-29T10:45:02.309 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:45:02.309 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:02.309 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:45:02.326623+0000 2024-03-29T10:45:02.309 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:45:02.326623+0000 2024-03-29T10:45:02.310 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:02.310 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:02.310 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:45:02.310 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:45:02.310 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:45:02.310 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:02.314 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.327+0000 7f41f9624640 1 Processor -- start 2024-03-29T10:45:02.314 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.327+0000 7f41f9624640 1 -- start start 2024-03-29T10:45:02.314 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.327+0000 7f41f9624640 10 monclient: init 2024-03-29T10:45:02.314 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.327+0000 7f41f9624640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:02.314 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.327+0000 7f41f9624640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:45:02.314 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.327+0000 7f41f9624640 10 monclient(hunting): picked mon.noname-c con 0x7f41ec02b4c0 addr v1:172.21.15.179:6789/0 2024-03-29T10:45:02.314 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.327+0000 7f41f9624640 10 monclient(hunting): picked mon.noname-b con 0x7f41ec00b8a0 addr v1:172.21.15.57:6790/0 2024-03-29T10:45:02.315 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.327+0000 7f41f9624640 10 monclient(hunting): picked mon.noname-a con 0x7f41ec02a5a0 addr v1:172.21.15.57:6789/0 2024-03-29T10:45:02.315 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.327+0000 7f41f9624640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ec02c5c0 con 0x7f41ec02a5a0 2024-03-29T10:45:02.315 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.327+0000 7f41f9624640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ec0c11a0 con 0x7f41ec02b4c0 2024-03-29T10:45:02.315 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.327+0000 7f41f9624640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ec02afa0 con 0x7f41ec00b8a0 2024-03-29T10:45:02.315 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.327+0000 7f41f9624640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:02.315 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.327+0000 7f41f9624640 10 monclient(hunting): authenticate will time out at 1254.503982s 2024-03-29T10:45:02.315 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.329+0000 7f41f8e23640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f41ec02a5a0 0x7f41ec0c6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44850/0 (socket says 172.21.15.57:44850) 2024-03-29T10:45:02.315 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.329+0000 7f41f8e23640 1 -- 172.21.15.57:0/1507996473 learned_addr learned my addr 172.21.15.57:0/1507996473 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:45:02.315 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.329+0000 7f41f2ffd640 1 -- 172.21.15.57:0/1507996473 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347279874 0 0) 0x7f41ec02c5c0 con 0x7f41ec02a5a0 2024-03-29T10:45:02.315 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.329+0000 7f41f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:02.316 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.329+0000 7f41f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:02.316 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.329+0000 7f41f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:02.316 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.329+0000 7f41f2ffd640 10 monclient(hunting): my global_id is 4404 2024-03-29T10:45:02.316 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.329+0000 7f41f2ffd640 1 -- 172.21.15.57:0/1507996473 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41e4003520 con 0x7f41ec02a5a0 2024-03-29T10:45:02.316 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.330+0000 7f41f2ffd640 1 -- 172.21.15.57:0/1507996473 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1671208515 0 0) 0x7f41e4003520 con 0x7f41ec02a5a0 2024-03-29T10:45:02.316 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.330+0000 7f41f2ffd640 1 -- 172.21.15.57:0/1507996473 >> v1:172.21.15.57:6790/0 conn(0x7f41ec00b8a0 legacy=0x7f41ec0c0a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:02.316 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.331+0000 7f41f2ffd640 1 -- 172.21.15.57:0/1507996473 >> v1:172.21.15.179:6789/0 conn(0x7f41ec02b4c0 legacy=0x7f41ec006890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:02.317 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.332+0000 7f41f2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:45:02.318 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.332+0000 7f41f2ffd640 1 monclient: found mon.noname-a 2024-03-29T10:45:02.318 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.332+0000 7f41f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:02.318 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.332+0000 7f41f2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:45:02.318 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.332+0000 7f41f2ffd640 1 -- 172.21.15.57:0/1507996473 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ec0329c0 con 0x7f41ec02a5a0 2024-03-29T10:45:02.318 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.332+0000 7f41f2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:45:02.318 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.332+0000 7f41f2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:45:02.318 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.332+0000 7f41f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:02.318 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.332+0000 7f41f2ffd640 1 -- 172.21.15.57:0/1507996473 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f41e0001690 con 0x7f41ec02a5a0 2024-03-29T10:45:02.318 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.332+0000 7f41f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:02.319 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.332+0000 7f41f2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:02.319 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.332+0000 7f41f2ffd640 10 monclient: dump: 2024-03-29T10:45:02.319 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:02.319 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:02.319 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:02.319 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:02.319 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:02.319 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:02.319 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:02.319 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:02.319 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:02.319 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:02.320 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.332+0000 7f41f9624640 5 monclient: authenticate success, global_id 4404 2024-03-29T10:45:02.320 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.332+0000 7f41f9624640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:02.320 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.333+0000 7f41f2ffd640 1 -- 172.21.15.57:0/1507996473 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41e0001860 con 0x7f41ec02a5a0 2024-03-29T10:45:02.320 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.333+0000 7f41f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:02.320 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.333+0000 7f41f9624640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:02.320 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.333+0000 7f41f9624640 4 set_mon_vals no callback set 2024-03-29T10:45:02.320 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.334+0000 7f41f9624640 10 monclient: shutdown 2024-03-29T10:45:02.320 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.334+0000 7f41f9624640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:02.320 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.334+0000 7f41f2ffd640 1 -- 172.21.15.57:0/1507996473 <== mon.0 v1:172.21.15.57:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f41e0001cb0 con 0x7f41ec02a5a0 2024-03-29T10:45:02.320 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.334+0000 7f41f9624640 1 -- 172.21.15.57:0/1507996473 >> v1:172.21.15.57:6789/0 conn(0x7f41ec02a5a0 legacy=0x7f41ec0c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:02.320 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.334+0000 7f41f2ffd640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:45:02.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.336+0000 7f41f9624640 1 -- 172.21.15.57:0/1507996473 shutdown_connections 2024-03-29T10:45:02.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.336+0000 7f41f9624640 1 -- 172.21.15.57:0/1507996473 >> 172.21.15.57:0/1507996473 conn(0x7f41ec0058f0 msgr2=0x7f41ec005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:02.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.336+0000 7f41f9624640 1 -- 172.21.15.57:0/1507996473 shutdown_connections 2024-03-29T10:45:02.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.336+0000 7f41f9624640 1 -- 172.21.15.57:0/1507996473 wait complete. 2024-03-29T10:45:02.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.337+0000 7f41f9624640 10 monclient: build_initial_monmap 2024-03-29T10:45:02.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.337+0000 7f41f9624640 10 monclient: monmap: 2024-03-29T10:45:02.322 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:45:02.322 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:02.322 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:45:02.322 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:45:02.322 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:02.322 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:02.322 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:45:02.322 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:45:02.322 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:45:02.323 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:02.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.337+0000 7f41f9624640 1 librados: starting msgr at 2024-03-29T10:45:02.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.337+0000 7f41f9624640 1 librados: starting objecter 2024-03-29T10:45:02.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.338+0000 7f41f9624640 1 Processor -- start 2024-03-29T10:45:02.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.341+0000 7f41f9624640 1 -- start start 2024-03-29T10:45:02.324 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.341+0000 7f41f9624640 1 librados: setting wanted keys 2024-03-29T10:45:02.324 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.341+0000 7f41f9624640 1 librados: calling monclient init 2024-03-29T10:45:02.324 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.341+0000 7f41f9624640 10 monclient: init 2024-03-29T10:45:02.324 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.341+0000 7f41f9624640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:02.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.341+0000 7f41f9624640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:45:02.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.342+0000 7f41f9624640 10 monclient(hunting): picked mon.noname-a con 0x7f41ec00b8a0 addr v1:172.21.15.57:6789/0 2024-03-29T10:45:02.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.342+0000 7f41f9624640 10 monclient(hunting): picked mon.noname-b con 0x7f41ec02a5a0 addr v1:172.21.15.179:6789/0 2024-03-29T10:45:02.328 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.342+0000 7f41f3fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f41ec00b8a0 0x7f41ec0c3d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:49960/0 (socket says 172.21.15.57:49960) 2024-03-29T10:45:02.328 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.345+0000 7f41f8e23640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f41ec02b4c0 0x7f41ec16b9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:37719/0 (socket says 172.21.15.57:37719) 2024-03-29T10:45:02.328 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.345+0000 7f41f9624640 10 monclient(hunting): picked mon.noname-c con 0x7f41ec02b4c0 addr v1:172.21.15.57:6790/0 2024-03-29T10:45:02.328 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.345+0000 7f41f9624640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ec0c3670 con 0x7f41ec00b8a0 2024-03-29T10:45:02.328 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.345+0000 7f41f9624640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ec0c3950 con 0x7f41ec02a5a0 2024-03-29T10:45:02.328 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.345+0000 7f41f9624640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ec0c6070 con 0x7f41ec02b4c0 2024-03-29T10:45:02.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.345+0000 7f41f9624640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:02.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.345+0000 7f41f9624640 10 monclient(hunting): authenticate will time out at 1254.521590s 2024-03-29T10:45:02.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.345+0000 7f41f3fff640 1 -- 172.21.15.57:0/3352010392 learned_addr learned my addr 172.21.15.57:0/3352010392 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:45:02.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.345+0000 7f41f0ff9640 10 client.?.objecter ms_handle_connect 0x7f41ec00b8a0 2024-03-29T10:45:02.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.345+0000 7f41f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:02.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.345+0000 7f41f0ff9640 10 client.?.objecter ms_handle_connect 0x7f41ec02b4c0 2024-03-29T10:45:02.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.345+0000 7f41f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:02.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.345+0000 7f41f0ff9640 1 -- 172.21.15.57:0/3352010392 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464118157 0 0) 0x7f41ec0c6070 con 0x7f41ec02b4c0 2024-03-29T10:45:02.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.345+0000 7f41f0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:45:02.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.345+0000 7f41f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:02.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.345+0000 7f41f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:02.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.345+0000 7f41f0ff9640 10 monclient(hunting): my global_id is 4436 2024-03-29T10:45:02.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.345+0000 7f41f0ff9640 1 -- 172.21.15.57:0/3352010392 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41d8004eb0 con 0x7f41ec02b4c0 2024-03-29T10:45:02.331 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.346+0000 7f41f0ff9640 1 -- 172.21.15.57:0/3352010392 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225049096 0 0) 0x7f41ec0c3670 con 0x7f41ec00b8a0 2024-03-29T10:45:02.331 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.346+0000 7f41f0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:45:02.331 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.346+0000 7f41f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:02.331 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.346+0000 7f41f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:02.331 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.346+0000 7f41f0ff9640 10 monclient(hunting): my global_id is 4407 2024-03-29T10:45:02.331 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.346+0000 7f41f0ff9640 1 -- 172.21.15.57:0/3352010392 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41ec0c6070 con 0x7f41ec00b8a0 2024-03-29T10:45:02.331 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.346+0000 7f41f0ff9640 1 -- 172.21.15.57:0/3352010392 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 295370328 0 0) 0x7f41ec0c6070 con 0x7f41ec00b8a0 2024-03-29T10:45:02.331 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.346+0000 7f41f0ff9640 1 -- 172.21.15.57:0/3352010392 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41ec0c3670 con 0x7f41ec00b8a0 2024-03-29T10:45:02.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.346+0000 7f41f0ff9640 1 -- 172.21.15.57:0/3352010392 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f41e8001830 con 0x7f41ec00b8a0 2024-03-29T10:45:02.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.346+0000 7f41f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:02.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.346+0000 7f41f0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:02.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.346+0000 7f41f0ff9640 10 monclient(hunting): dump: 2024-03-29T10:45:02.332 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:02.332 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:02.332 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:02.332 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:02.332 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:02.332 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:02.333 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:02.333 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:02.333 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:02.333 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:02.333 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.346+0000 7f41f0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:02.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.348+0000 7f41f0ff9640 1 -- 172.21.15.57:0/3352010392 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3080034468 0 0) 0x7f41ec0c3670 con 0x7f41ec00b8a0 2024-03-29T10:45:02.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.348+0000 7f41f0ff9640 1 -- 172.21.15.57:0/3352010392 >> v1:172.21.15.57:6790/0 conn(0x7f41ec02b4c0 legacy=0x7f41ec16b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:02.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.348+0000 7f41f0ff9640 1 -- 172.21.15.57:0/3352010392 >> v1:172.21.15.179:6789/0 conn(0x7f41ec02a5a0 legacy=0x7f41ec0c14d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:45:02.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.348+0000 7f41f0ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:45:02.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.348+0000 7f41f0ff9640 1 monclient: found mon.a 2024-03-29T10:45:02.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.348+0000 7f41f0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:02.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.348+0000 7f41f0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:45:02.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.348+0000 7f41f0ff9640 1 -- 172.21.15.57:0/3352010392 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ec0c6fb0 con 0x7f41ec00b8a0 2024-03-29T10:45:02.338 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.348+0000 7f41f0ff9640 10 monclient: _finish_auth 0 2024-03-29T10:45:02.338 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.349+0000 7f41f0ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:45:02.338 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.349+0000 7f41f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:02.338 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.349+0000 7f41f9624640 5 monclient: authenticate success, global_id 4407 2024-03-29T10:45:02.338 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.349+0000 7f41f9624640 10 monclient: _renew_subs 2024-03-29T10:45:02.338 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.349+0000 7f41f9624640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:45:02.338 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.349+0000 7f41f9624640 1 -- 172.21.15.57:0/3352010392 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41ec0c4480 con 0x7f41ec00b8a0 2024-03-29T10:45:02.338 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.349+0000 7f41f9624640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:02.338 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.349+0000 7f41f9624640 10 monclient: _renew_subs 2024-03-29T10:45:02.338 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.349+0000 7f41f9624640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:45:02.338 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.349+0000 7f41f9624640 1 -- 172.21.15.57:0/3352010392 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41ec0c1dc0 con 0x7f41ec00b8a0 2024-03-29T10:45:02.338 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.349+0000 7f41f9624640 1 librados: init done 2024-03-29T10:45:02.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.349+0000 7f41f0ff9640 1 -- 172.21.15.57:0/3352010392 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41e8001d40 con 0x7f41ec00b8a0 2024-03-29T10:45:02.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.349+0000 7f41f0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:02.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.349+0000 7f41f0ff9640 1 -- 172.21.15.57:0/3352010392 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f41e8004aa0 con 0x7f41ec00b8a0 2024-03-29T10:45:02.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.349+0000 7f41f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:02.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.349+0000 7f41f0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:02.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.349+0000 7f41f0ff9640 10 monclient: dump: 2024-03-29T10:45:02.339 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:02.339 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:02.339 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:02.339 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:02.339 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:02.339 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:02.339 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:02.340 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:02.340 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:02.340 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:02.340 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.349+0000 7f41f1ffb640 4 set_mon_vals no callback set 2024-03-29T10:45:02.340 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.350+0000 7f41f9624640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:02.340 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.350+0000 7f41f9624640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:02.340 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.350+0000 7f41f9624640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:45:02.340 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.350+0000 7f41f9624640 1 -- 172.21.15.57:0/3352010392 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d4005080 con 0x7f41ec00b8a0 2024-03-29T10:45:02.340 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.351+0000 7f41f0ff9640 1 -- 172.21.15.57:0/3352010392 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f41e801aa70 con 0x7f41ec00b8a0 2024-03-29T10:45:02.340 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.352+0000 7f41f0ff9640 1 -- 172.21.15.57:0/3352010392 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f41e8084ab0 con 0x7f41ec00b8a0 2024-03-29T10:45:02.340 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.352+0000 7f41f0ff9640 10 client.4407.objecter ms_dispatch 0x7f41f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:02.340 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.352+0000 7f41f0ff9640 3 client.4407.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:02.341 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.352+0000 7f41f0ff9640 3 client.4407.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:02.341 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.352+0000 7f41f0ff9640 20 client.4407.objecter dump_active .. 0 homeless 2024-03-29T10:45:02.341 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.354+0000 7f41f37fe640 10 monclient: get_auth_request con 0x7f41d806dd50 auth_method 0 2024-03-29T10:45:02.341 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.354+0000 7f41f0ff9640 10 client.4407.objecter ms_handle_connect 0x7f41d806dd50 2024-03-29T10:45:02.344 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.361+0000 7f41f0ff9640 1 -- 172.21.15.57:0/3352010392 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f41e80501c0 con 0x7f41ec00b8a0 2024-03-29T10:45:02.344 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.361+0000 7f41f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:02.344 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.361+0000 7f41f0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:02.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.678+0000 7f41f9624640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:02.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.678+0000 7f41f9624640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:02.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.678+0000 7f41f9624640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:45:02.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.678+0000 7f41f9624640 1 -- 172.21.15.57:0/3352010392 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41d4005330 con 0x7f41ec00b8a0 2024-03-29T10:45:02.662 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.680+0000 7f41f0ff9640 1 -- 172.21.15.57:0/3352010392 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f41e80522c0 con 0x7f41ec00b8a0 2024-03-29T10:45:02.662 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.680+0000 7f41f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:02.662 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.680+0000 7f41f0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:02.663 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:45:02.663 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:02.666 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.683+0000 7f41f9624640 10 librados: watch_flush enter 2024-03-29T10:45:02.666 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.683+0000 7f41f9624640 10 librados: watch_flush exit 2024-03-29T10:45:02.666 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.683+0000 7f41f9624640 20 client.4407.objecter shutdown clearing up homeless session... 2024-03-29T10:45:02.666 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.683+0000 7f41f9624640 10 client.4407.objecter successfully canceled tick 2024-03-29T10:45:02.666 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.683+0000 7f41f9624640 1 -- 172.21.15.57:0/3352010392 >> v1:172.21.15.179:6816/2698295936 conn(0x7f41d806dd50 legacy=0x7f41d8070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:02.666 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.683+0000 7f41f9624640 10 monclient: shutdown 2024-03-29T10:45:02.666 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.683+0000 7f41f9624640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:02.666 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.683+0000 7f41f9624640 1 -- 172.21.15.57:0/3352010392 >> v1:172.21.15.57:6789/0 conn(0x7f41ec00b8a0 legacy=0x7f41ec0c3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:02.667 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.685+0000 7f41f9624640 1 -- 172.21.15.57:0/3352010392 shutdown_connections 2024-03-29T10:45:02.667 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.685+0000 7f41f9624640 1 -- 172.21.15.57:0/3352010392 >> 172.21.15.57:0/3352010392 conn(0x7f41ec0058f0 msgr2=0x7f41ec034a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:02.668 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.685+0000 7f41f9624640 1 -- 172.21.15.57:0/3352010392 shutdown_connections 2024-03-29T10:45:02.668 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.685+0000 7f41f9624640 1 -- 172.21.15.57:0/3352010392 wait complete. 2024-03-29T10:45:02.668 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:02.685+0000 7f41f9624640 1 librados: shutdown 2024-03-29T10:45:02.683 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:09.491 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:09.641 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.658+0000 7f5da93a8640 10 monclient: get_monmap_and_config 2024-03-29T10:45:09.641 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.658+0000 7f5da93a8640 10 monclient: build_initial_monmap 2024-03-29T10:45:09.641 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.658+0000 7f5da93a8640 10 monclient: monmap: 2024-03-29T10:45:09.641 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:45:09.642 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:09.642 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:45:09.659642+0000 2024-03-29T10:45:09.642 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:45:09.659642+0000 2024-03-29T10:45:09.642 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:09.642 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:09.642 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:45:09.642 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:45:09.642 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:45:09.642 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:09.643 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.660+0000 7f5da93a8640 1 Processor -- start 2024-03-29T10:45:09.643 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.660+0000 7f5da93a8640 1 -- start start 2024-03-29T10:45:09.643 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.660+0000 7f5da93a8640 10 monclient: init 2024-03-29T10:45:09.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.661+0000 7f5da93a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:09.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.661+0000 7f5da93a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:45:09.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.661+0000 7f5da93a8640 10 monclient(hunting): picked mon.noname-c con 0x7f5da406cac0 addr v1:172.21.15.179:6789/0 2024-03-29T10:45:09.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.661+0000 7f5da93a8640 10 monclient(hunting): picked mon.noname-b con 0x7f5da406cea0 addr v1:172.21.15.57:6790/0 2024-03-29T10:45:09.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.661+0000 7f5da93a8640 10 monclient(hunting): picked mon.noname-a con 0x7f5da4145e80 addr v1:172.21.15.57:6789/0 2024-03-29T10:45:09.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.661+0000 7f5da93a8640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da40ac850 con 0x7f5da4145e80 2024-03-29T10:45:09.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.661+0000 7f5da93a8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da4070470 con 0x7f5da406cac0 2024-03-29T10:45:09.645 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.661+0000 7f5da93a8640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da4072600 con 0x7f5da406cea0 2024-03-29T10:45:09.645 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.661+0000 7f5da93a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:09.645 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.661+0000 7f5da93a8640 10 monclient(hunting): authenticate will time out at 1261.837795s 2024-03-29T10:45:09.645 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.661+0000 7f5da37fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f5da4145e80 0x7f5da414db10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:59952/0 (socket says 172.21.15.57:59952) 2024-03-29T10:45:09.645 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.661+0000 7f5da37fe640 1 -- 172.21.15.57:0/3407540433 learned_addr learned my addr 172.21.15.57:0/3407540433 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:45:09.649 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.662+0000 7f5da1ffb640 1 -- 172.21.15.57:0/3407540433 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112501492 0 0) 0x7f5da40ac850 con 0x7f5da4145e80 2024-03-29T10:45:09.649 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.662+0000 7f5da1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:45:09.649 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.662+0000 7f5da1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:09.649 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.662+0000 7f5da1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:09.649 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.662+0000 7f5da1ffb640 10 monclient(hunting): my global_id is 4410 2024-03-29T10:45:09.650 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.662+0000 7f5da1ffb640 1 -- 172.21.15.57:0/3407540433 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d8c003520 con 0x7f5da4145e80 2024-03-29T10:45:09.650 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.662+0000 7f5da1ffb640 1 -- 172.21.15.57:0/3407540433 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1846962043 0 0) 0x7f5da4072600 con 0x7f5da406cea0 2024-03-29T10:45:09.650 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.662+0000 7f5da1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:45:09.650 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.662+0000 7f5da1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:09.650 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.662+0000 7f5da1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:09.650 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.662+0000 7f5da1ffb640 10 monclient(hunting): my global_id is 4439 2024-03-29T10:45:09.650 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.662+0000 7f5da1ffb640 1 -- 172.21.15.57:0/3407540433 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5da40ac850 con 0x7f5da406cea0 2024-03-29T10:45:09.650 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.663+0000 7f5da1ffb640 1 -- 172.21.15.57:0/3407540433 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334546714 0 0) 0x7f5da4070470 con 0x7f5da406cac0 2024-03-29T10:45:09.650 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.663+0000 7f5da1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:45:09.650 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.663+0000 7f5da1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:09.650 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.663+0000 7f5da1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:09.650 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.663+0000 7f5da1ffb640 10 monclient(hunting): my global_id is 4408 2024-03-29T10:45:09.650 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.663+0000 7f5da1ffb640 1 -- 172.21.15.57:0/3407540433 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5da4072600 con 0x7f5da406cac0 2024-03-29T10:45:09.650 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.663+0000 7f5da1ffb640 1 -- 172.21.15.57:0/3407540433 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3861782135 0 0) 0x7f5d8c003520 con 0x7f5da4145e80 2024-03-29T10:45:09.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.663+0000 7f5da1ffb640 1 -- 172.21.15.57:0/3407540433 >> v1:172.21.15.57:6790/0 conn(0x7f5da406cea0 legacy=0x7f5da4145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:09.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.663+0000 7f5da1ffb640 1 -- 172.21.15.57:0/3407540433 >> v1:172.21.15.179:6789/0 conn(0x7f5da406cac0 legacy=0x7f5da4071ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:09.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.663+0000 7f5da1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:45:09.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.663+0000 7f5da1ffb640 1 monclient: found mon.noname-a 2024-03-29T10:45:09.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.663+0000 7f5da1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:09.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.663+0000 7f5da1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:45:09.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.663+0000 7f5da1ffb640 1 -- 172.21.15.57:0/3407540433 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da41462d0 con 0x7f5da4145e80 2024-03-29T10:45:09.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.663+0000 7f5da1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:45:09.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.663+0000 7f5da1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:45:09.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.663+0000 7f5da1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:09.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.663+0000 7f5da1ffb640 1 -- 172.21.15.57:0/3407540433 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f5d9c001a10 con 0x7f5da4145e80 2024-03-29T10:45:09.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.663+0000 7f5da1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:09.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.663+0000 7f5da1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:09.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.663+0000 7f5da1ffb640 10 monclient: dump: 2024-03-29T10:45:09.651 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:09.652 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:09.652 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:09.652 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:09.652 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:09.652 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:09.652 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:09.652 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:09.652 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:09.652 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:09.652 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.663+0000 7f5da93a8640 5 monclient: authenticate success, global_id 4410 2024-03-29T10:45:09.652 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.663+0000 7f5da93a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:09.652 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.664+0000 7f5da1ffb640 1 -- 172.21.15.57:0/3407540433 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d9c001c30 con 0x7f5da4145e80 2024-03-29T10:45:09.652 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.664+0000 7f5da1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:09.652 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.664+0000 7f5da1ffb640 1 -- 172.21.15.57:0/3407540433 <== mon.0 v1:172.21.15.57:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f5d9c001fc0 con 0x7f5da4145e80 2024-03-29T10:45:09.652 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.664+0000 7f5da1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:09.653 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.664+0000 7f5da1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:09.653 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.664+0000 7f5da1ffb640 10 monclient: dump: 2024-03-29T10:45:09.653 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:09.653 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:09.653 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:09.653 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:09.653 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:09.653 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:09.653 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:09.653 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:09.653 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:09.653 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:09.653 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.664+0000 7f5da93a8640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:09.654 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.664+0000 7f5da93a8640 4 set_mon_vals no callback set 2024-03-29T10:45:09.654 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.664+0000 7f5da93a8640 10 monclient: shutdown 2024-03-29T10:45:09.654 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.665+0000 7f5da93a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:09.654 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.665+0000 7f5da93a8640 1 -- 172.21.15.57:0/3407540433 >> v1:172.21.15.57:6789/0 conn(0x7f5da4145e80 legacy=0x7f5da414db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:09.654 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.665+0000 7f5da93a8640 1 -- 172.21.15.57:0/3407540433 shutdown_connections 2024-03-29T10:45:09.654 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.665+0000 7f5da93a8640 1 -- 172.21.15.57:0/3407540433 >> 172.21.15.57:0/3407540433 conn(0x7f5da40675b0 msgr2=0x7f5da40b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:09.654 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.665+0000 7f5da93a8640 1 -- 172.21.15.57:0/3407540433 shutdown_connections 2024-03-29T10:45:09.654 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.665+0000 7f5da93a8640 1 -- 172.21.15.57:0/3407540433 wait complete. 2024-03-29T10:45:09.654 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.666+0000 7f5da93a8640 10 monclient: build_initial_monmap 2024-03-29T10:45:09.654 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.666+0000 7f5da93a8640 10 monclient: monmap: 2024-03-29T10:45:09.654 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:45:09.654 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:09.654 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:45:09.655 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:45:09.655 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:09.655 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:09.655 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:45:09.655 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:45:09.655 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:45:09.655 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:09.655 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.666+0000 7f5da93a8640 1 librados: starting msgr at 2024-03-29T10:45:09.655 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.666+0000 7f5da93a8640 1 librados: starting objecter 2024-03-29T10:45:09.655 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.667+0000 7f5da93a8640 1 Processor -- start 2024-03-29T10:45:09.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.667+0000 7f5da93a8640 1 -- start start 2024-03-29T10:45:09.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.667+0000 7f5da93a8640 1 librados: setting wanted keys 2024-03-29T10:45:09.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.667+0000 7f5da93a8640 1 librados: calling monclient init 2024-03-29T10:45:09.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.667+0000 7f5da93a8640 10 monclient: init 2024-03-29T10:45:09.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.668+0000 7f5da93a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:09.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.668+0000 7f5da93a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:45:09.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.668+0000 7f5da93a8640 10 monclient(hunting): picked mon.noname-c con 0x7f5da406cac0 addr v1:172.21.15.57:6790/0 2024-03-29T10:45:09.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.668+0000 7f5da93a8640 10 monclient(hunting): picked mon.noname-b con 0x7f5da406cea0 addr v1:172.21.15.179:6789/0 2024-03-29T10:45:09.662 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.668+0000 7f5da93a8640 10 monclient(hunting): picked mon.noname-a con 0x7f5da4145e80 addr v1:172.21.15.57:6789/0 2024-03-29T10:45:09.662 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.668+0000 7f5da93a8640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da40aad90 con 0x7f5da4145e80 2024-03-29T10:45:09.662 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.668+0000 7f5da37fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f5da4145e80 0x7f5da40aa400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:60598/0 (socket says 172.21.15.57:60598) 2024-03-29T10:45:09.662 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.668+0000 7f5da37fe640 1 -- 172.21.15.57:0/3130836802 learned_addr learned my addr 172.21.15.57:0/3130836802 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:45:09.662 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.668+0000 7f5da93a8640 1 -- 172.21.15.57:0/3130836802 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da40ae0e0 con 0x7f5da406cea0 2024-03-29T10:45:09.662 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.668+0000 7f5da93a8640 1 -- 172.21.15.57:0/3130836802 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da40aea10 con 0x7f5da406cac0 2024-03-29T10:45:09.662 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.668+0000 7f5da93a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:09.662 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.668+0000 7f5da93a8640 10 monclient(hunting): authenticate will time out at 1261.844789s 2024-03-29T10:45:09.662 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.668+0000 7f5d8b7fe640 10 client.?.objecter ms_handle_connect 0x7f5da4145e80 2024-03-29T10:45:09.662 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.668+0000 7f5d8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:09.662 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.668+0000 7f5d8b7fe640 10 client.?.objecter ms_handle_connect 0x7f5da406cea0 2024-03-29T10:45:09.662 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.668+0000 7f5d8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:09.662 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.669+0000 7f5d8b7fe640 10 client.?.objecter ms_handle_connect 0x7f5da406cac0 2024-03-29T10:45:09.662 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.669+0000 7f5d8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:09.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.669+0000 7f5d8b7fe640 1 -- 172.21.15.57:0/3130836802 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463373668 0 0) 0x7f5da40ae0e0 con 0x7f5da406cea0 2024-03-29T10:45:09.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.669+0000 7f5d8b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:09.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.669+0000 7f5d8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:09.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.669+0000 7f5d8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:09.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.669+0000 7f5d8b7fe640 10 monclient(hunting): my global_id is 4411 2024-03-29T10:45:09.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.669+0000 7f5d8b7fe640 1 -- 172.21.15.57:0/3130836802 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d90004eb0 con 0x7f5da406cea0 2024-03-29T10:45:09.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.669+0000 7f5d8b7fe640 1 -- 172.21.15.57:0/3130836802 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177501666 0 0) 0x7f5da40aad90 con 0x7f5da4145e80 2024-03-29T10:45:09.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.669+0000 7f5d8b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:09.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.669+0000 7f5d8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:09.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.669+0000 7f5d8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:09.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.669+0000 7f5d8b7fe640 10 monclient(hunting): my global_id is 4413 2024-03-29T10:45:09.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.669+0000 7f5d8b7fe640 1 -- 172.21.15.57:0/3130836802 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5da40ae0e0 con 0x7f5da4145e80 2024-03-29T10:45:09.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.670+0000 7f5d8b7fe640 1 -- 172.21.15.57:0/3130836802 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1474603956 0 0) 0x7f5d90004eb0 con 0x7f5da406cea0 2024-03-29T10:45:09.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.670+0000 7f5d8b7fe640 1 -- 172.21.15.57:0/3130836802 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5da40aad90 con 0x7f5da406cea0 2024-03-29T10:45:09.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.670+0000 7f5d8b7fe640 1 -- 172.21.15.57:0/3130836802 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f5d98001930 con 0x7f5da406cea0 2024-03-29T10:45:09.664 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.670+0000 7f5d8b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:09.664 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.670+0000 7f5d8b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:09.664 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.670+0000 7f5d8b7fe640 10 monclient(hunting): dump: 2024-03-29T10:45:09.664 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:09.664 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:09.664 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:09.664 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:09.664 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:09.664 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:09.664 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:09.664 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:09.664 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:09.664 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:09.664 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.670+0000 7f5d8b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:09.664 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.670+0000 7f5d8b7fe640 1 -- 172.21.15.57:0/3130836802 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 195240787 0 0) 0x7f5da40ae0e0 con 0x7f5da4145e80 2024-03-29T10:45:09.664 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.670+0000 7f5d8b7fe640 1 -- 172.21.15.57:0/3130836802 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d90004eb0 con 0x7f5da4145e80 2024-03-29T10:45:09.665 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.670+0000 7f5d8b7fe640 1 -- 172.21.15.57:0/3130836802 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f5d9c001f40 con 0x7f5da4145e80 2024-03-29T10:45:09.665 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.670+0000 7f5d8b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:09.665 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.670+0000 7f5d8b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:09.665 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.670+0000 7f5d8b7fe640 10 monclient(hunting): dump: 2024-03-29T10:45:09.665 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:09.665 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:09.665 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:09.665 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:09.665 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:09.665 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:09.665 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:09.665 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:09.665 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:09.665 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:09.666 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.670+0000 7f5d8b7fe640 1 -- 172.21.15.57:0/3130836802 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344762906 0 0) 0x7f5da40aea10 con 0x7f5da406cac0 2024-03-29T10:45:09.666 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.670+0000 7f5d8b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:09.666 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.670+0000 7f5d8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:09.666 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.670+0000 7f5d8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:09.666 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.670+0000 7f5d8b7fe640 10 monclient(hunting): my global_id is 4442 2024-03-29T10:45:09.666 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.671+0000 7f5d8b7fe640 1 -- 172.21.15.57:0/3130836802 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5da40ae0e0 con 0x7f5da406cac0 2024-03-29T10:45:09.666 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.671+0000 7f5d8b7fe640 1 -- 172.21.15.57:0/3130836802 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 148444220 0 0) 0x7f5da40aad90 con 0x7f5da406cea0 2024-03-29T10:45:09.666 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.671+0000 7f5d8b7fe640 1 -- 172.21.15.57:0/3130836802 >> v1:172.21.15.57:6790/0 conn(0x7f5da406cac0 legacy=0x7f5da40ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:09.666 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.671+0000 7f5d8b7fe640 1 -- 172.21.15.57:0/3130836802 >> v1:172.21.15.57:6789/0 conn(0x7f5da4145e80 legacy=0x7f5da40aa400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:09.667 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.671+0000 7f5d8b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:45:09.667 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.671+0000 7f5d8b7fe640 1 monclient: found mon.b 2024-03-29T10:45:09.667 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.671+0000 7f5d8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:09.667 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.671+0000 7f5d8b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:45:09.667 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.671+0000 7f5d8b7fe640 1 -- 172.21.15.57:0/3130836802 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da40aecf0 con 0x7f5da406cea0 2024-03-29T10:45:09.667 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.671+0000 7f5d8b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:45:09.667 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.671+0000 7f5d8b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:45:09.667 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.671+0000 7f5d8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:09.667 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.671+0000 7f5da93a8640 5 monclient: authenticate success, global_id 4411 2024-03-29T10:45:09.667 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.671+0000 7f5da93a8640 10 monclient: _renew_subs 2024-03-29T10:45:09.667 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.671+0000 7f5da93a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:45:09.667 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.671+0000 7f5da93a8640 1 -- 172.21.15.57:0/3130836802 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da40aefd0 con 0x7f5da406cea0 2024-03-29T10:45:09.667 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.671+0000 7f5da93a8640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:09.667 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.671+0000 7f5da93a8640 10 monclient: _renew_subs 2024-03-29T10:45:09.667 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.671+0000 7f5da93a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:45:09.668 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.671+0000 7f5da93a8640 1 -- 172.21.15.57:0/3130836802 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da40aab10 con 0x7f5da406cea0 2024-03-29T10:45:09.668 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.671+0000 7f5da93a8640 1 librados: init done 2024-03-29T10:45:09.668 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.671+0000 7f5d8b7fe640 1 -- 172.21.15.57:0/3130836802 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d98001a90 con 0x7f5da406cea0 2024-03-29T10:45:09.668 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.671+0000 7f5d8b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:09.668 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.671+0000 7f5d8b7fe640 1 -- 172.21.15.57:0/3130836802 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f5d980038c0 con 0x7f5da406cea0 2024-03-29T10:45:09.668 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.672+0000 7f5d8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:09.668 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.672+0000 7f5d8b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:09.668 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.672+0000 7f5d8b7fe640 10 monclient: dump: 2024-03-29T10:45:09.668 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:09.668 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:09.668 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:09.668 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:09.669 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:09.669 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:09.669 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:09.669 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:09.669 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:09.669 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:09.669 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.672+0000 7f5da17fa640 4 set_mon_vals no callback set 2024-03-29T10:45:09.669 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.673+0000 7f5da93a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:09.669 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.673+0000 7f5da93a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:09.669 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.673+0000 7f5da93a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:45:09.669 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.673+0000 7f5da93a8640 1 -- 172.21.15.57:0/3130836802 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d84005080 con 0x7f5da406cea0 2024-03-29T10:45:09.669 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.674+0000 7f5d8b7fe640 1 -- 172.21.15.57:0/3130836802 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f5d98019890 con 0x7f5da406cea0 2024-03-29T10:45:09.669 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.675+0000 7f5d8b7fe640 1 -- 172.21.15.57:0/3130836802 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f5d980837d0 con 0x7f5da406cea0 2024-03-29T10:45:09.670 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.675+0000 7f5d8b7fe640 10 client.4411.objecter ms_dispatch 0x7f5da4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:09.670 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.675+0000 7f5d8b7fe640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:09.670 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.675+0000 7f5d8b7fe640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:09.670 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.675+0000 7f5d8b7fe640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-29T10:45:09.670 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.675+0000 7f5da2ffd640 10 monclient: get_auth_request con 0x7f5d9006df90 auth_method 0 2024-03-29T10:45:09.670 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.676+0000 7f5d8b7fe640 10 client.4411.objecter ms_handle_connect 0x7f5d9006df90 2024-03-29T10:45:09.670 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.681+0000 7f5d8b7fe640 1 -- 172.21.15.57:0/3130836802 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5d9804eee0 con 0x7f5da406cea0 2024-03-29T10:45:09.670 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.681+0000 7f5d8b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:09.671 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:09.681+0000 7f5d8b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:10.084 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:10.101+0000 7f5da93a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:10.084 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:10.101+0000 7f5da93a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:10.084 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:10.101+0000 7f5da93a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:45:10.084 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:10.101+0000 7f5da93a8640 1 -- 172.21.15.57:0/3130836802 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d84005330 con 0x7f5da406cea0 2024-03-29T10:45:10.085 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:10.102+0000 7f5d8b7fe640 1 -- 172.21.15.57:0/3130836802 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f5d98050fe0 con 0x7f5da406cea0 2024-03-29T10:45:10.085 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:10.102+0000 7f5d8b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:10.085 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:10.102+0000 7f5d8b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:10.085 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:45:10.085 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:10.089 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:10.107+0000 7f5da93a8640 10 librados: watch_flush enter 2024-03-29T10:45:10.090 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:10.107+0000 7f5da93a8640 10 librados: watch_flush exit 2024-03-29T10:45:10.090 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:10.107+0000 7f5da93a8640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-29T10:45:10.090 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:10.107+0000 7f5da93a8640 10 client.4411.objecter successfully canceled tick 2024-03-29T10:45:10.090 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:10.107+0000 7f5da93a8640 1 -- 172.21.15.57:0/3130836802 >> v1:172.21.15.179:6816/2698295936 conn(0x7f5d9006df90 legacy=0x7f5d90070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:10.090 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:10.107+0000 7f5da93a8640 10 monclient: shutdown 2024-03-29T10:45:10.091 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:10.107+0000 7f5da93a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:10.091 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:10.107+0000 7f5da93a8640 1 -- 172.21.15.57:0/3130836802 >> v1:172.21.15.179:6789/0 conn(0x7f5da406cea0 legacy=0x7f5da40a9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:10.091 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:10.107+0000 7f5da93a8640 1 -- 172.21.15.57:0/3130836802 shutdown_connections 2024-03-29T10:45:10.091 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:10.107+0000 7f5da93a8640 1 -- 172.21.15.57:0/3130836802 >> 172.21.15.57:0/3130836802 conn(0x7f5da40675b0 msgr2=0x7f5da40b2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:10.091 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:10.107+0000 7f5da93a8640 1 -- 172.21.15.57:0/3130836802 shutdown_connections 2024-03-29T10:45:10.091 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:10.108+0000 7f5da93a8640 1 -- 172.21.15.57:0/3130836802 wait complete. 2024-03-29T10:45:10.091 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:10.108+0000 7f5da93a8640 1 librados: shutdown 2024-03-29T10:45:10.105 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:16.912 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:17.043 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.061+0000 7fe7964dd640 10 monclient: get_monmap_and_config 2024-03-29T10:45:17.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.061+0000 7fe7964dd640 10 monclient: build_initial_monmap 2024-03-29T10:45:17.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.061+0000 7fe7964dd640 10 monclient: monmap: 2024-03-29T10:45:17.044 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:45:17.044 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:17.044 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:45:17.061840+0000 2024-03-29T10:45:17.044 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:45:17.061840+0000 2024-03-29T10:45:17.044 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:17.044 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:17.044 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:45:17.044 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:45:17.044 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:45:17.044 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:17.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.062+0000 7fe7964dd640 1 Processor -- start 2024-03-29T10:45:17.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.062+0000 7fe7964dd640 1 -- start start 2024-03-29T10:45:17.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.062+0000 7fe7964dd640 10 monclient: init 2024-03-29T10:45:17.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.063+0000 7fe7964dd640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:17.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.063+0000 7fe7964dd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:45:17.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.063+0000 7fe7964dd640 10 monclient(hunting): picked mon.noname-c con 0x7fe7900b0860 addr v1:172.21.15.179:6789/0 2024-03-29T10:45:17.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.063+0000 7fe7964dd640 10 monclient(hunting): picked mon.noname-a con 0x7fe790072450 addr v1:172.21.15.57:6789/0 2024-03-29T10:45:17.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.064+0000 7fe7964dd640 10 monclient(hunting): picked mon.noname-b con 0x7fe79013ee40 addr v1:172.21.15.57:6790/0 2024-03-29T10:45:17.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.064+0000 7fe7964dd640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7900ad840 con 0x7fe790072450 2024-03-29T10:45:17.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.064+0000 7fe7964dd640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79013c830 con 0x7fe7900b0860 2024-03-29T10:45:17.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.064+0000 7fe7964dd640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79013c9c0 con 0x7fe79013ee40 2024-03-29T10:45:17.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.064+0000 7fe7964dd640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:17.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.064+0000 7fe7964dd640 10 monclient(hunting): authenticate will time out at 1269.240167s 2024-03-29T10:45:17.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.064+0000 7fe794a7a640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7fe79013ee40 0x7fe79013f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:43316/0 (socket says 172.21.15.57:43316) 2024-03-29T10:45:17.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.064+0000 7fe794a7a640 1 -- 172.21.15.57:0/668455845 learned_addr learned my addr 172.21.15.57:0/668455845 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:45:17.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.064+0000 7fe78ffff640 1 --1- 172.21.15.57:0/668455845 >> v1:172.21.15.179:6789/0 conn(0x7fe7900b0860 0x7fe7900acfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:36964/0 (socket says 172.21.15.57:36964) 2024-03-29T10:45:17.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.064+0000 7fe78effd640 1 -- 172.21.15.57:0/668455845 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210323585 0 0) 0x7fe79013c9c0 con 0x7fe79013ee40 2024-03-29T10:45:17.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.064+0000 7fe78effd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:17.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.064+0000 7fe78effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:17.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.064+0000 7fe78effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:17.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.064+0000 7fe78effd640 10 monclient(hunting): my global_id is 4445 2024-03-29T10:45:17.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.064+0000 7fe78effd640 1 -- 172.21.15.57:0/668455845 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe778003520 con 0x7fe79013ee40 2024-03-29T10:45:17.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.064+0000 7fe78effd640 1 -- 172.21.15.57:0/668455845 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3231143465 0 0) 0x7fe778003520 con 0x7fe79013ee40 2024-03-29T10:45:17.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.064+0000 7fe78effd640 1 -- 172.21.15.57:0/668455845 >> v1:172.21.15.179:6789/0 conn(0x7fe7900b0860 legacy=0x7fe7900acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:17.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.064+0000 7fe78effd640 1 -- 172.21.15.57:0/668455845 >> v1:172.21.15.57:6789/0 conn(0x7fe790072450 legacy=0x7fe79013c120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:17.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.067+0000 7fe78effd640 10 monclient: _finish_hunting 0 2024-03-29T10:45:17.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.067+0000 7fe78effd640 1 monclient: found mon.noname-b 2024-03-29T10:45:17.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.067+0000 7fe78effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:17.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.067+0000 7fe78effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:45:17.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.067+0000 7fe78effd640 1 -- 172.21.15.57:0/668455845 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe79013f970 con 0x7fe79013ee40 2024-03-29T10:45:17.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.067+0000 7fe78effd640 10 monclient: _finish_auth 0 2024-03-29T10:45:17.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.067+0000 7fe78effd640 10 monclient: _check_auth_tickets 2024-03-29T10:45:17.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.067+0000 7fe78effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:17.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.067+0000 7fe78effd640 1 -- 172.21.15.57:0/668455845 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fe788001a10 con 0x7fe79013ee40 2024-03-29T10:45:17.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.067+0000 7fe78effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:17.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.067+0000 7fe78effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:17.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.067+0000 7fe78effd640 10 monclient: dump: 2024-03-29T10:45:17.053 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:17.053 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:17.053 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:17.053 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:17.053 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:17.053 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:17.053 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:17.053 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:17.053 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:17.053 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:17.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.067+0000 7fe7964dd640 5 monclient: authenticate success, global_id 4445 2024-03-29T10:45:17.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.067+0000 7fe7964dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:17.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.067+0000 7fe78effd640 1 -- 172.21.15.57:0/668455845 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe788001ca0 con 0x7fe79013ee40 2024-03-29T10:45:17.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.067+0000 7fe78effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:17.054 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.067+0000 7fe78effd640 1 -- 172.21.15.57:0/668455845 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fe7880020b0 con 0x7fe79013ee40 2024-03-29T10:45:17.054 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.067+0000 7fe78effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:17.054 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.067+0000 7fe78effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:17.054 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.067+0000 7fe78effd640 10 monclient: dump: 2024-03-29T10:45:17.054 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:17.054 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:17.054 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:17.054 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:17.054 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:17.054 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:17.054 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:17.054 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:17.054 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:17.054 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:17.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.067+0000 7fe7964dd640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:17.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.067+0000 7fe7964dd640 4 set_mon_vals no callback set 2024-03-29T10:45:17.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.068+0000 7fe7964dd640 10 monclient: shutdown 2024-03-29T10:45:17.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.068+0000 7fe7964dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:17.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.068+0000 7fe7964dd640 1 -- 172.21.15.57:0/668455845 >> v1:172.21.15.57:6790/0 conn(0x7fe79013ee40 legacy=0x7fe79013f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:17.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.068+0000 7fe7964dd640 1 -- 172.21.15.57:0/668455845 shutdown_connections 2024-03-29T10:45:17.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.068+0000 7fe7964dd640 1 -- 172.21.15.57:0/668455845 >> 172.21.15.57:0/668455845 conn(0x7fe79006b660 msgr2=0x7fe79006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:17.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.068+0000 7fe7964dd640 1 -- 172.21.15.57:0/668455845 shutdown_connections 2024-03-29T10:45:17.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.068+0000 7fe7964dd640 1 -- 172.21.15.57:0/668455845 wait complete. 2024-03-29T10:45:17.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.069+0000 7fe7964dd640 10 monclient: build_initial_monmap 2024-03-29T10:45:17.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.070+0000 7fe7964dd640 10 monclient: monmap: 2024-03-29T10:45:17.055 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:45:17.055 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:17.055 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:45:17.055 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:45:17.056 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:17.056 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:17.056 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:45:17.056 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:45:17.056 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:45:17.056 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:17.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.070+0000 7fe7964dd640 1 librados: starting msgr at 2024-03-29T10:45:17.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.070+0000 7fe7964dd640 1 librados: starting objecter 2024-03-29T10:45:17.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.070+0000 7fe7964dd640 1 Processor -- start 2024-03-29T10:45:17.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.070+0000 7fe7964dd640 1 -- start start 2024-03-29T10:45:17.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.070+0000 7fe7964dd640 1 librados: setting wanted keys 2024-03-29T10:45:17.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.070+0000 7fe7964dd640 1 librados: calling monclient init 2024-03-29T10:45:17.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.070+0000 7fe7964dd640 10 monclient: init 2024-03-29T10:45:17.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.071+0000 7fe7964dd640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:17.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.071+0000 7fe7964dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:45:17.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.071+0000 7fe7964dd640 10 monclient(hunting): picked mon.noname-a con 0x7fe790072450 addr v1:172.21.15.57:6789/0 2024-03-29T10:45:17.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.071+0000 7fe7964dd640 10 monclient(hunting): picked mon.noname-c con 0x7fe7900b0860 addr v1:172.21.15.57:6790/0 2024-03-29T10:45:17.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.071+0000 7fe7964dd640 10 monclient(hunting): picked mon.noname-b con 0x7fe79013ee40 addr v1:172.21.15.179:6789/0 2024-03-29T10:45:17.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.071+0000 7fe7964dd640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe790142030 con 0x7fe790072450 2024-03-29T10:45:17.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.071+0000 7fe7964dd640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe790143100 con 0x7fe79013ee40 2024-03-29T10:45:17.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.071+0000 7fe7964dd640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7901355a0 con 0x7fe7900b0860 2024-03-29T10:45:17.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.071+0000 7fe7964dd640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:17.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.071+0000 7fe7964dd640 10 monclient(hunting): authenticate will time out at 1269.247730s 2024-03-29T10:45:17.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.071+0000 7fe794a7a640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fe79013ee40 0x7fe790141490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:37112/0 (socket says 172.21.15.57:37112) 2024-03-29T10:45:17.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.071+0000 7fe794a7a640 1 -- 172.21.15.57:0/813408548 learned_addr learned my addr 172.21.15.57:0/813408548 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:45:17.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.072+0000 7fe78cff9640 10 client.?.objecter ms_handle_connect 0x7fe79013ee40 2024-03-29T10:45:17.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.072+0000 7fe78cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:17.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.072+0000 7fe78cff9640 10 client.?.objecter ms_handle_connect 0x7fe7900b0860 2024-03-29T10:45:17.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.072+0000 7fe78cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:17.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.072+0000 7fe78cff9640 1 -- 172.21.15.57:0/813408548 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1274048997 0 0) 0x7fe7901355a0 con 0x7fe7900b0860 2024-03-29T10:45:17.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.072+0000 7fe78cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:45:17.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.072+0000 7fe78cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:17.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.072+0000 7fe78cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:17.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.072+0000 7fe78cff9640 10 monclient(hunting): my global_id is 4448 2024-03-29T10:45:17.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.072+0000 7fe78cff9640 1 -- 172.21.15.57:0/813408548 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe77c004eb0 con 0x7fe7900b0860 2024-03-29T10:45:17.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.072+0000 7fe78cff9640 1 -- 172.21.15.57:0/813408548 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3122262141 0 0) 0x7fe77c004eb0 con 0x7fe7900b0860 2024-03-29T10:45:17.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.072+0000 7fe78cff9640 1 -- 172.21.15.57:0/813408548 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7901355a0 con 0x7fe7900b0860 2024-03-29T10:45:17.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.072+0000 7fe78cff9640 1 -- 172.21.15.57:0/813408548 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fe784001510 con 0x7fe7900b0860 2024-03-29T10:45:17.066 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.073+0000 7fe78cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:17.066 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.073+0000 7fe78cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:17.066 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.073+0000 7fe78cff9640 10 monclient(hunting): dump: 2024-03-29T10:45:17.066 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:17.066 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:17.066 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:17.066 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:17.066 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:17.066 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:17.066 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:17.066 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:17.066 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:17.066 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:17.066 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.073+0000 7fe78cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:17.067 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.073+0000 7fe78cff9640 1 -- 172.21.15.57:0/813408548 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 830039574 0 0) 0x7fe7901355a0 con 0x7fe7900b0860 2024-03-29T10:45:17.067 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.073+0000 7fe78cff9640 1 -- 172.21.15.57:0/813408548 >> v1:172.21.15.179:6789/0 conn(0x7fe79013ee40 legacy=0x7fe790141490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:17.067 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.073+0000 7fe78cff9640 1 -- 172.21.15.57:0/813408548 >> v1:172.21.15.57:6789/0 conn(0x7fe790072450 legacy=0x7fe7901debf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:17.067 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.075+0000 7fe78cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:45:17.067 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.075+0000 7fe78cff9640 1 monclient: found mon.c 2024-03-29T10:45:17.067 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.075+0000 7fe78cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:17.067 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.075+0000 7fe78cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:45:17.067 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.075+0000 7fe78cff9640 1 -- 172.21.15.57:0/813408548 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe790141ba0 con 0x7fe7900b0860 2024-03-29T10:45:17.067 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.075+0000 7fe78cff9640 10 monclient: _finish_auth 0 2024-03-29T10:45:17.067 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.075+0000 7fe78cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:45:17.067 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.075+0000 7fe78cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:17.067 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.075+0000 7fe7964dd640 5 monclient: authenticate success, global_id 4448 2024-03-29T10:45:17.067 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.075+0000 7fe7964dd640 10 monclient: _renew_subs 2024-03-29T10:45:17.067 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.075+0000 7fe7964dd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:45:17.067 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.075+0000 7fe7964dd640 1 -- 172.21.15.57:0/813408548 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe790134b00 con 0x7fe7900b0860 2024-03-29T10:45:17.068 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.075+0000 7fe7964dd640 10 client.4448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:17.068 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.075+0000 7fe7964dd640 10 monclient: _renew_subs 2024-03-29T10:45:17.068 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.075+0000 7fe7964dd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:45:17.068 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.075+0000 7fe7964dd640 1 -- 172.21.15.57:0/813408548 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7900b2840 con 0x7fe7900b0860 2024-03-29T10:45:17.068 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.075+0000 7fe7964dd640 1 librados: init done 2024-03-29T10:45:17.068 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.075+0000 7fe78cff9640 1 -- 172.21.15.57:0/813408548 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe784001aa0 con 0x7fe7900b0860 2024-03-29T10:45:17.068 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.075+0000 7fe78cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:17.068 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.075+0000 7fe78e7fc640 4 set_mon_vals no callback set 2024-03-29T10:45:17.068 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.076+0000 7fe78cff9640 1 -- 172.21.15.57:0/813408548 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fe784003910 con 0x7fe7900b0860 2024-03-29T10:45:17.068 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.076+0000 7fe78cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:17.068 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.076+0000 7fe78cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:17.068 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.076+0000 7fe78cff9640 10 monclient: dump: 2024-03-29T10:45:17.068 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:17.068 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:17.068 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:17.069 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:17.069 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:17.069 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:17.069 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:17.069 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:17.069 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:17.069 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:17.069 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.077+0000 7fe7964dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:17.069 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.077+0000 7fe7964dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:17.069 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.077+0000 7fe7964dd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:45:17.069 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.077+0000 7fe7964dd640 1 -- 172.21.15.57:0/813408548 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe770005080 con 0x7fe7900b0860 2024-03-29T10:45:17.069 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.078+0000 7fe78cff9640 1 -- 172.21.15.57:0/813408548 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fe7840198e0 con 0x7fe7900b0860 2024-03-29T10:45:17.069 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.079+0000 7fe78cff9640 1 -- 172.21.15.57:0/813408548 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7fe784083830 con 0x7fe7900b0860 2024-03-29T10:45:17.069 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.079+0000 7fe78cff9640 10 client.4448.objecter ms_dispatch 0x7fe790000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:17.069 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.079+0000 7fe78cff9640 3 client.4448.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:17.070 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.079+0000 7fe78cff9640 3 client.4448.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:17.070 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.079+0000 7fe78cff9640 20 client.4448.objecter dump_active .. 0 homeless 2024-03-29T10:45:17.070 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.079+0000 7fe78ffff640 10 monclient: get_auth_request con 0x7fe77c06dcb0 auth_method 0 2024-03-29T10:45:17.070 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.080+0000 7fe78cff9640 10 client.4448.objecter ms_handle_connect 0x7fe77c06dcb0 2024-03-29T10:45:17.070 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.087+0000 7fe78cff9640 1 -- 172.21.15.57:0/813408548 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe78404ef40 con 0x7fe7900b0860 2024-03-29T10:45:17.070 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.087+0000 7fe78cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:17.071 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.087+0000 7fe78cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:17.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.406+0000 7fe7964dd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:17.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.406+0000 7fe7964dd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:17.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.406+0000 7fe7964dd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:45:17.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.406+0000 7fe7964dd640 1 -- 172.21.15.57:0/813408548 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe770005330 con 0x7fe7900b0860 2024-03-29T10:45:17.390 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.408+0000 7fe78cff9640 1 -- 172.21.15.57:0/813408548 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7fe784051040 con 0x7fe7900b0860 2024-03-29T10:45:17.390 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.408+0000 7fe78cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:17.390 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.408+0000 7fe78cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:17.391 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:45:17.391 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:17.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.410+0000 7fe7964dd640 10 librados: watch_flush enter 2024-03-29T10:45:17.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.410+0000 7fe7964dd640 10 librados: watch_flush exit 2024-03-29T10:45:17.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.410+0000 7fe7964dd640 20 client.4448.objecter shutdown clearing up homeless session... 2024-03-29T10:45:17.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.410+0000 7fe7964dd640 10 client.4448.objecter successfully canceled tick 2024-03-29T10:45:17.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.410+0000 7fe7964dd640 1 -- 172.21.15.57:0/813408548 >> v1:172.21.15.179:6816/2698295936 conn(0x7fe77c06dcb0 legacy=0x7fe77c070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:17.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.410+0000 7fe7964dd640 10 monclient: shutdown 2024-03-29T10:45:17.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.410+0000 7fe7964dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:17.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.410+0000 7fe7964dd640 1 -- 172.21.15.57:0/813408548 >> v1:172.21.15.57:6790/0 conn(0x7fe7900b0860 legacy=0x7fe790140d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:17.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.411+0000 7fe7964dd640 1 -- 172.21.15.57:0/813408548 shutdown_connections 2024-03-29T10:45:17.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.411+0000 7fe7964dd640 1 -- 172.21.15.57:0/813408548 >> 172.21.15.57:0/813408548 conn(0x7fe79006b660 msgr2=0x7fe79013bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:17.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.411+0000 7fe7964dd640 1 -- 172.21.15.57:0/813408548 shutdown_connections 2024-03-29T10:45:17.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.411+0000 7fe7964dd640 1 -- 172.21.15.57:0/813408548 wait complete. 2024-03-29T10:45:17.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:17.411+0000 7fe7964dd640 1 librados: shutdown 2024-03-29T10:45:17.408 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:24.216 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:24.352 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.367+0000 7f7209258640 10 monclient: get_monmap_and_config 2024-03-29T10:45:24.352 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.367+0000 7f7209258640 10 monclient: build_initial_monmap 2024-03-29T10:45:24.352 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.367+0000 7f7209258640 10 monclient: monmap: 2024-03-29T10:45:24.352 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:45:24.353 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:24.353 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:45:24.369385+0000 2024-03-29T10:45:24.353 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:45:24.369385+0000 2024-03-29T10:45:24.353 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:24.353 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:24.353 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:45:24.353 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:45:24.353 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:45:24.353 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:24.353 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.368+0000 7f7209258640 1 Processor -- start 2024-03-29T10:45:24.353 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.368+0000 7f7209258640 1 -- start start 2024-03-29T10:45:24.353 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.368+0000 7f7209258640 10 monclient: init 2024-03-29T10:45:24.355 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.369+0000 7f7209258640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:24.355 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.369+0000 7f7209258640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:45:24.355 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.370+0000 7f7209258640 10 monclient(hunting): picked mon.noname-a con 0x7f72041422b0 addr v1:172.21.15.57:6789/0 2024-03-29T10:45:24.355 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.370+0000 7f7209258640 10 monclient(hunting): picked mon.noname-c con 0x7f7204142690 addr v1:172.21.15.179:6789/0 2024-03-29T10:45:24.355 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.370+0000 7f7209258640 10 monclient(hunting): picked mon.noname-b con 0x7f72040b17d0 addr v1:172.21.15.57:6790/0 2024-03-29T10:45:24.355 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.370+0000 7f7209258640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7204142c80 con 0x7f72041422b0 2024-03-29T10:45:24.355 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.370+0000 7f7209258640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72040aae40 con 0x7f7204142690 2024-03-29T10:45:24.355 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.370+0000 7f7209258640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f720413fbd0 con 0x7f72040b17d0 2024-03-29T10:45:24.356 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.370+0000 7f7209258640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:24.356 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.370+0000 7f7209258640 10 monclient(hunting): authenticate will time out at 1276.547300s 2024-03-29T10:45:24.356 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.370+0000 7f720359e640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f72040b17d0 0x7f72040b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:47934/0 (socket says 172.21.15.57:47934) 2024-03-29T10:45:24.356 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.370+0000 7f720359e640 1 -- 172.21.15.57:0/1673869243 learned_addr learned my addr 172.21.15.57:0/1673869243 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:45:24.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 1 -- 172.21.15.57:0/1673869243 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4084573446 0 0) 0x7f720413fbd0 con 0x7f72040b17d0 2024-03-29T10:45:24.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:45:24.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:24.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:24.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 10 monclient(hunting): my global_id is 4451 2024-03-29T10:45:24.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 1 -- 172.21.15.57:0/1673869243 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71ec003520 con 0x7f72040b17d0 2024-03-29T10:45:24.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 1 -- 172.21.15.57:0/1673869243 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 123586973 0 0) 0x7f71ec003520 con 0x7f72040b17d0 2024-03-29T10:45:24.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 1 -- 172.21.15.57:0/1673869243 >> v1:172.21.15.179:6789/0 conn(0x7f7204142690 legacy=0x7f72040b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:45:24.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 1 -- 172.21.15.57:0/1673869243 >> v1:172.21.15.57:6789/0 conn(0x7f72041422b0 legacy=0x7f720413f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:24.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:45:24.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 1 monclient: found mon.noname-b 2024-03-29T10:45:24.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:24.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:45:24.362 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 1 -- 172.21.15.57:0/1673869243 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7204151940 con 0x7f72040b17d0 2024-03-29T10:45:24.362 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 10 monclient: _finish_auth 0 2024-03-29T10:45:24.362 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:45:24.362 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:24.362 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 1 -- 172.21.15.57:0/1673869243 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f71fc001a10 con 0x7f72040b17d0 2024-03-29T10:45:24.362 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:24.362 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:24.362 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 10 monclient: dump: 2024-03-29T10:45:24.362 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:24.362 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:24.363 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:24.363 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:24.363 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:24.363 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:24.363 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:24.363 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:24.364 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:24.364 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:24.364 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 1 -- 172.21.15.57:0/1673869243 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71fc001b70 con 0x7f72040b17d0 2024-03-29T10:45:24.364 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:24.364 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 1 -- 172.21.15.57:0/1673869243 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f71fc001fc0 con 0x7f72040b17d0 2024-03-29T10:45:24.364 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:24.364 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:24.365 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.372+0000 7f7201d9b640 10 monclient: dump: 2024-03-29T10:45:24.365 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:24.365 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:24.365 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:24.365 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:24.365 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:24.365 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:24.365 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:24.365 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:24.365 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:24.366 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:24.366 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.373+0000 7f7209258640 5 monclient: authenticate success, global_id 4451 2024-03-29T10:45:24.366 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.373+0000 7f7209258640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:24.366 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.373+0000 7f7209258640 4 set_mon_vals no callback set 2024-03-29T10:45:24.366 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.374+0000 7f7209258640 10 monclient: shutdown 2024-03-29T10:45:24.366 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.374+0000 7f7209258640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:24.366 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.374+0000 7f7209258640 1 -- 172.21.15.57:0/1673869243 >> v1:172.21.15.57:6790/0 conn(0x7f72040b17d0 legacy=0x7f72040b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:24.367 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.374+0000 7f7209258640 1 -- 172.21.15.57:0/1673869243 shutdown_connections 2024-03-29T10:45:24.367 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.374+0000 7f7209258640 1 -- 172.21.15.57:0/1673869243 >> 172.21.15.57:0/1673869243 conn(0x7f7204066a00 msgr2=0x7f7204066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:24.367 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.375+0000 7f7209258640 1 -- 172.21.15.57:0/1673869243 shutdown_connections 2024-03-29T10:45:24.367 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.375+0000 7f7209258640 1 -- 172.21.15.57:0/1673869243 wait complete. 2024-03-29T10:45:24.367 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.376+0000 7f7209258640 10 monclient: build_initial_monmap 2024-03-29T10:45:24.367 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.376+0000 7f7209258640 10 monclient: monmap: 2024-03-29T10:45:24.367 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:45:24.367 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:24.367 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:45:24.367 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:45:24.367 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:24.368 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:24.368 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:45:24.368 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:45:24.368 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:45:24.368 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:24.368 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.377+0000 7f7209258640 1 librados: starting msgr at 2024-03-29T10:45:24.368 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.377+0000 7f7209258640 1 librados: starting objecter 2024-03-29T10:45:24.368 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.377+0000 7f7209258640 1 Processor -- start 2024-03-29T10:45:24.368 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.377+0000 7f7209258640 1 -- start start 2024-03-29T10:45:24.368 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.377+0000 7f7209258640 1 librados: setting wanted keys 2024-03-29T10:45:24.368 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.377+0000 7f7209258640 1 librados: calling monclient init 2024-03-29T10:45:24.368 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.377+0000 7f7209258640 10 monclient: init 2024-03-29T10:45:24.373 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.378+0000 7f7209258640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:24.373 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.378+0000 7f7209258640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:45:24.373 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.378+0000 7f7209258640 10 monclient(hunting): picked mon.noname-c con 0x7f72040b17d0 addr v1:172.21.15.57:6790/0 2024-03-29T10:45:24.373 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.378+0000 7f7209258640 10 monclient(hunting): picked mon.noname-b con 0x7f72041422b0 addr v1:172.21.15.179:6789/0 2024-03-29T10:45:24.373 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.378+0000 7f7202d9d640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f72040b17d0 0x7f72041df0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:49288/0 (socket says 172.21.15.57:49288) 2024-03-29T10:45:24.373 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.378+0000 7f720259c640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f72041422b0 0x7f720412ecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:39378/0 (socket says 172.21.15.57:39378) 2024-03-29T10:45:24.373 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.378+0000 7f7209258640 10 monclient(hunting): picked mon.noname-a con 0x7f7204142690 addr v1:172.21.15.57:6789/0 2024-03-29T10:45:24.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.378+0000 7f7209258640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f720406ee60 con 0x7f7204142690 2024-03-29T10:45:24.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.378+0000 7f7209258640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72041df9e0 con 0x7f72041422b0 2024-03-29T10:45:24.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.378+0000 7f7209258640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f720406de90 con 0x7f72040b17d0 2024-03-29T10:45:24.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.378+0000 7f7209258640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:24.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.378+0000 7f7209258640 10 monclient(hunting): authenticate will time out at 1276.555978s 2024-03-29T10:45:24.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.379+0000 7f7202d9d640 1 -- 172.21.15.57:0/768182441 learned_addr learned my addr 172.21.15.57:0/768182441 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:45:24.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.379+0000 7f71eb7fe640 10 client.?.objecter ms_handle_connect 0x7f72040b17d0 2024-03-29T10:45:24.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.379+0000 7f71eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:24.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.379+0000 7f71eb7fe640 10 client.?.objecter ms_handle_connect 0x7f72041422b0 2024-03-29T10:45:24.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.379+0000 7f71eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:24.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.379+0000 7f71eb7fe640 1 -- 172.21.15.57:0/768182441 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586384480 0 0) 0x7f720406de90 con 0x7f72040b17d0 2024-03-29T10:45:24.374 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.379+0000 7f71eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:24.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.379+0000 7f71eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:24.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.379+0000 7f71eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:24.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.379+0000 7f71eb7fe640 10 monclient(hunting): my global_id is 4454 2024-03-29T10:45:24.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.379+0000 7f71eb7fe640 1 -- 172.21.15.57:0/768182441 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71f0004eb0 con 0x7f72040b17d0 2024-03-29T10:45:24.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.379+0000 7f71eb7fe640 1 -- 172.21.15.57:0/768182441 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177330073 0 0) 0x7f72041df9e0 con 0x7f72041422b0 2024-03-29T10:45:24.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.379+0000 7f71eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:24.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.379+0000 7f71eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:24.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.379+0000 7f71eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:24.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.379+0000 7f71eb7fe640 10 monclient(hunting): my global_id is 4420 2024-03-29T10:45:24.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.379+0000 7f71eb7fe640 1 -- 172.21.15.57:0/768182441 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f720406de90 con 0x7f72041422b0 2024-03-29T10:45:24.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.380+0000 7f71eb7fe640 1 -- 172.21.15.57:0/768182441 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2882279629 0 0) 0x7f71f0004eb0 con 0x7f72040b17d0 2024-03-29T10:45:24.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.380+0000 7f71eb7fe640 1 -- 172.21.15.57:0/768182441 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72041df9e0 con 0x7f72040b17d0 2024-03-29T10:45:24.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.380+0000 7f71eb7fe640 1 -- 172.21.15.57:0/768182441 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f71f4001960 con 0x7f72040b17d0 2024-03-29T10:45:24.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.380+0000 7f71eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:24.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.380+0000 7f71eb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:24.375 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.380+0000 7f71eb7fe640 10 monclient(hunting): dump: 2024-03-29T10:45:24.376 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:24.376 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:24.376 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:24.376 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:24.376 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:24.376 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:24.376 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:24.376 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:24.376 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:24.376 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:24.376 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.380+0000 7f71eb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:24.376 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.380+0000 7f71eb7fe640 1 -- 172.21.15.57:0/768182441 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3234490184 0 0) 0x7f720406de90 con 0x7f72041422b0 2024-03-29T10:45:24.376 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.380+0000 7f71eb7fe640 1 -- 172.21.15.57:0/768182441 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71f0004eb0 con 0x7f72041422b0 2024-03-29T10:45:24.376 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.380+0000 7f71eb7fe640 1 -- 172.21.15.57:0/768182441 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f71f8003ee0 con 0x7f72041422b0 2024-03-29T10:45:24.376 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.380+0000 7f71eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:24.376 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.380+0000 7f71eb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:24.377 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.380+0000 7f71eb7fe640 10 monclient(hunting): dump: 2024-03-29T10:45:24.377 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:24.377 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:24.377 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:24.377 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:24.377 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:24.377 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:24.377 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:24.377 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:24.377 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:24.377 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:24.377 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.380+0000 7f71eb7fe640 1 -- 172.21.15.57:0/768182441 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 367174579 0 0) 0x7f72041df9e0 con 0x7f72040b17d0 2024-03-29T10:45:24.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.380+0000 7f71eb7fe640 1 -- 172.21.15.57:0/768182441 >> v1:172.21.15.179:6789/0 conn(0x7f72041422b0 legacy=0x7f720412ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:24.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.380+0000 7f71eb7fe640 1 -- 172.21.15.57:0/768182441 >> v1:172.21.15.57:6789/0 conn(0x7f7204142690 legacy=0x7f7204131b60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:24.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.382+0000 7f71eb7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:45:24.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.382+0000 7f71eb7fe640 1 monclient: found mon.c 2024-03-29T10:45:24.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.382+0000 7f71eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:24.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.382+0000 7f71eb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:45:24.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.382+0000 7f71eb7fe640 1 -- 172.21.15.57:0/768182441 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f720413bba0 con 0x7f72040b17d0 2024-03-29T10:45:24.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.382+0000 7f71eb7fe640 10 monclient: _finish_auth 0 2024-03-29T10:45:24.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.382+0000 7f71eb7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:45:24.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.382+0000 7f71eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:24.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.382+0000 7f7209258640 5 monclient: authenticate success, global_id 4454 2024-03-29T10:45:24.378 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.382+0000 7f7209258640 10 monclient: _renew_subs 2024-03-29T10:45:24.379 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.382+0000 7f7209258640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:45:24.379 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.382+0000 7f7209258640 1 -- 172.21.15.57:0/768182441 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f720413be50 con 0x7f72040b17d0 2024-03-29T10:45:24.379 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.382+0000 7f7209258640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:24.379 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.382+0000 7f71eb7fe640 1 -- 172.21.15.57:0/768182441 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71f40035f0 con 0x7f72040b17d0 2024-03-29T10:45:24.379 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.382+0000 7f71eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:24.379 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.382+0000 7f71eb7fe640 1 -- 172.21.15.57:0/768182441 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f71f4003940 con 0x7f72040b17d0 2024-03-29T10:45:24.379 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.382+0000 7f71eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:24.379 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.382+0000 7f71eb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:24.379 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.382+0000 7f71eb7fe640 10 monclient: dump: 2024-03-29T10:45:24.379 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:24.379 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:24.379 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:24.379 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:24.379 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:24.380 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:24.380 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:24.380 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:24.380 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:24.380 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:24.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.382+0000 7f7209258640 10 monclient: _renew_subs 2024-03-29T10:45:24.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.382+0000 7f7209258640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:45:24.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.382+0000 7f7209258640 1 -- 172.21.15.57:0/768182441 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f720412f5a0 con 0x7f72040b17d0 2024-03-29T10:45:24.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.382+0000 7f7209258640 1 librados: init done 2024-03-29T10:45:24.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.382+0000 7f720159a640 4 set_mon_vals no callback set 2024-03-29T10:45:24.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.383+0000 7f7209258640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:24.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.383+0000 7f7209258640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:24.380 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.383+0000 7f7209258640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:45:24.381 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.383+0000 7f7209258640 1 -- 172.21.15.57:0/768182441 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7204075f10 con 0x7f72040b17d0 2024-03-29T10:45:24.381 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.384+0000 7f71eb7fe640 1 -- 172.21.15.57:0/768182441 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f71f4003aa0 con 0x7f72040b17d0 2024-03-29T10:45:24.381 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.384+0000 7f71eb7fe640 1 -- 172.21.15.57:0/768182441 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f71f4084a40 con 0x7f72040b17d0 2024-03-29T10:45:24.381 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.384+0000 7f71eb7fe640 10 client.4454.objecter ms_dispatch 0x7f7204000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:24.381 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.384+0000 7f71eb7fe640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:24.381 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.384+0000 7f71eb7fe640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:24.381 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.385+0000 7f71eb7fe640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-29T10:45:24.381 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.385+0000 7f720259c640 10 monclient: get_auth_request con 0x7f71f006df10 auth_method 0 2024-03-29T10:45:24.381 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.385+0000 7f71eb7fe640 10 client.4454.objecter ms_handle_connect 0x7f71f006df10 2024-03-29T10:45:24.382 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.392+0000 7f71eb7fe640 1 -- 172.21.15.57:0/768182441 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f71f4050320 con 0x7f72040b17d0 2024-03-29T10:45:24.382 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.392+0000 7f71eb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:24.382 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.392+0000 7f71eb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:24.693 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.709+0000 7f7209258640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:24.693 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.709+0000 7f7209258640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:24.693 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.709+0000 7f7209258640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:45:24.693 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.709+0000 7f7209258640 1 -- 172.21.15.57:0/768182441 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72041df7b0 con 0x7f72040b17d0 2024-03-29T10:45:24.693 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.709+0000 7f71eb7fe640 1 -- 172.21.15.57:0/768182441 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f71f4052250 con 0x7f72040b17d0 2024-03-29T10:45:24.693 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.709+0000 7f71eb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:24.693 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.709+0000 7f71eb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:24.694 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:45:24.694 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:24.696 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.712+0000 7f7209258640 10 librados: watch_flush enter 2024-03-29T10:45:24.697 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.713+0000 7f7209258640 10 librados: watch_flush exit 2024-03-29T10:45:24.697 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.713+0000 7f7209258640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-29T10:45:24.697 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.713+0000 7f7209258640 10 client.4454.objecter successfully canceled tick 2024-03-29T10:45:24.697 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.713+0000 7f7209258640 1 -- 172.21.15.57:0/768182441 >> v1:172.21.15.179:6816/2698295936 conn(0x7f71f006df10 legacy=0x7f71f00703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:24.697 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.713+0000 7f7209258640 10 monclient: shutdown 2024-03-29T10:45:24.697 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.713+0000 7f7209258640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:24.697 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.713+0000 7f7209258640 1 -- 172.21.15.57:0/768182441 >> v1:172.21.15.57:6790/0 conn(0x7f72040b17d0 legacy=0x7f72041df0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:24.697 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.713+0000 7f7209258640 1 -- 172.21.15.57:0/768182441 shutdown_connections 2024-03-29T10:45:24.697 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.713+0000 7f7209258640 1 -- 172.21.15.57:0/768182441 >> 172.21.15.57:0/768182441 conn(0x7f7204066a00 msgr2=0x7f72040b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:24.697 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.713+0000 7f7209258640 1 -- 172.21.15.57:0/768182441 shutdown_connections 2024-03-29T10:45:24.697 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.713+0000 7f7209258640 1 -- 172.21.15.57:0/768182441 wait complete. 2024-03-29T10:45:24.697 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:24.713+0000 7f7209258640 1 librados: shutdown 2024-03-29T10:45:24.713 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:31.519 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:31.672 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.687+0000 7f9729ed3640 10 monclient: get_monmap_and_config 2024-03-29T10:45:31.672 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.687+0000 7f9729ed3640 10 monclient: build_initial_monmap 2024-03-29T10:45:31.672 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.687+0000 7f9729ed3640 10 monclient: monmap: 2024-03-29T10:45:31.672 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:45:31.672 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:31.672 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:45:31.689112+0000 2024-03-29T10:45:31.672 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:45:31.689112+0000 2024-03-29T10:45:31.672 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:31.672 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:31.673 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:45:31.673 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:45:31.673 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:45:31.673 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:31.673 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.688+0000 7f9729ed3640 1 Processor -- start 2024-03-29T10:45:31.673 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.688+0000 7f9729ed3640 1 -- start start 2024-03-29T10:45:31.673 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.688+0000 7f9729ed3640 10 monclient: init 2024-03-29T10:45:31.676 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.689+0000 7f9729ed3640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:31.676 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.689+0000 7f9729ed3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:45:31.676 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.689+0000 7f9729ed3640 10 monclient(hunting): picked mon.noname-c con 0x7f97240ae250 addr v1:172.21.15.179:6789/0 2024-03-29T10:45:31.676 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.689+0000 7f9729ed3640 10 monclient(hunting): picked mon.noname-b con 0x7f972413f970 addr v1:172.21.15.57:6790/0 2024-03-29T10:45:31.676 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.689+0000 7f9729ed3640 10 monclient(hunting): picked mon.noname-a con 0x7f9724142800 addr v1:172.21.15.57:6789/0 2024-03-29T10:45:31.677 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.689+0000 7f9729ed3640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97240afe00 con 0x7f9724142800 2024-03-29T10:45:31.677 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.689+0000 7f9729ed3640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9724142c10 con 0x7f97240ae250 2024-03-29T10:45:31.677 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.689+0000 7f9729ed3640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97240afb20 con 0x7f972413f970 2024-03-29T10:45:31.677 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.689+0000 7f9729ed3640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:31.677 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.689+0000 7f9729ed3640 10 monclient(hunting): authenticate will time out at 1283.866584s 2024-03-29T10:45:31.680 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.694+0000 7f97237fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f97240ae250 0x7f972413f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:59085/0 (socket says 172.21.15.57:59085) 2024-03-29T10:45:31.680 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.694+0000 7f9722ffd640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f972413f970 0x7f97241420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:39511/0 (socket says 172.21.15.57:39511) 2024-03-29T10:45:31.680 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.694+0000 7f97237fe640 1 -- 172.21.15.57:0/878190617 learned_addr learned my addr 172.21.15.57:0/878190617 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:45:31.680 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.694+0000 7f97227fc640 1 -- 172.21.15.57:0/878190617 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630249204 0 0) 0x7f9724142c10 con 0x7f97240ae250 2024-03-29T10:45:31.680 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.694+0000 7f97227fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:31.680 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.694+0000 7f97227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:31.680 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.694+0000 7f97227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:31.680 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.694+0000 7f97227fc640 10 monclient(hunting): my global_id is 4423 2024-03-29T10:45:31.680 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.694+0000 7f97227fc640 1 -- 172.21.15.57:0/878190617 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f970c003520 con 0x7f97240ae250 2024-03-29T10:45:31.680 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.694+0000 7f97227fc640 1 -- 172.21.15.57:0/878190617 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187496910 0 0) 0x7f97240afb20 con 0x7f972413f970 2024-03-29T10:45:31.680 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.694+0000 7f97227fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:31.681 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.694+0000 7f97227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:31.681 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.694+0000 7f97227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:31.681 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.694+0000 7f97227fc640 10 monclient(hunting): my global_id is 4457 2024-03-29T10:45:31.681 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.694+0000 7f97227fc640 1 -- 172.21.15.57:0/878190617 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9724142c10 con 0x7f972413f970 2024-03-29T10:45:31.681 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.695+0000 7f97227fc640 1 -- 172.21.15.57:0/878190617 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 932061379 0 0) 0x7f970c003520 con 0x7f97240ae250 2024-03-29T10:45:31.681 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.695+0000 7f97227fc640 1 -- 172.21.15.57:0/878190617 >> v1:172.21.15.57:6790/0 conn(0x7f972413f970 legacy=0x7f97241420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:31.681 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.695+0000 7f97227fc640 1 -- 172.21.15.57:0/878190617 >> v1:172.21.15.57:6789/0 conn(0x7f9724142800 legacy=0x7f972414da10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:31.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.697+0000 7f97227fc640 10 monclient: _finish_hunting 0 2024-03-29T10:45:31.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.697+0000 7f97227fc640 1 monclient: found mon.noname-c 2024-03-29T10:45:31.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.697+0000 7f97227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:31.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.697+0000 7f97227fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:45:31.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.697+0000 7f97227fc640 1 -- 172.21.15.57:0/878190617 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97240aab80 con 0x7f97240ae250 2024-03-29T10:45:31.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.697+0000 7f97227fc640 10 monclient: _finish_auth 0 2024-03-29T10:45:31.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.697+0000 7f97227fc640 10 monclient: _check_auth_tickets 2024-03-29T10:45:31.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.697+0000 7f97227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:31.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.697+0000 7f97227fc640 1 -- 172.21.15.57:0/878190617 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f97180015c0 con 0x7f97240ae250 2024-03-29T10:45:31.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.697+0000 7f97227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:31.685 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.697+0000 7f97227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:31.685 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.697+0000 7f97227fc640 10 monclient: dump: 2024-03-29T10:45:31.685 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:31.685 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:31.685 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:31.685 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:31.685 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:31.685 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:31.685 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:31.685 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:31.685 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:31.685 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:31.685 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.697+0000 7f9729ed3640 5 monclient: authenticate success, global_id 4423 2024-03-29T10:45:31.685 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.697+0000 7f9729ed3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:31.685 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.697+0000 7f97227fc640 1 -- 172.21.15.57:0/878190617 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9718001720 con 0x7f97240ae250 2024-03-29T10:45:31.686 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.697+0000 7f97227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:31.686 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.697+0000 7f97227fc640 1 -- 172.21.15.57:0/878190617 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f9718001b70 con 0x7f97240ae250 2024-03-29T10:45:31.686 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.697+0000 7f97227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:31.686 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.697+0000 7f97227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:31.686 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.697+0000 7f97227fc640 10 monclient: dump: 2024-03-29T10:45:31.686 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:31.686 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:31.686 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:31.686 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:31.686 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:31.686 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:31.686 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:31.686 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:31.686 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:31.687 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:31.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.697+0000 7f9729ed3640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:31.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.697+0000 7f9729ed3640 4 set_mon_vals no callback set 2024-03-29T10:45:31.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.698+0000 7f9729ed3640 10 monclient: shutdown 2024-03-29T10:45:31.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.698+0000 7f9729ed3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:31.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.698+0000 7f9729ed3640 1 -- 172.21.15.57:0/878190617 >> v1:172.21.15.179:6789/0 conn(0x7f97240ae250 legacy=0x7f972413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:31.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.699+0000 7f9729ed3640 1 -- 172.21.15.57:0/878190617 shutdown_connections 2024-03-29T10:45:31.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.699+0000 7f9729ed3640 1 -- 172.21.15.57:0/878190617 >> 172.21.15.57:0/878190617 conn(0x7f97240b35e0 msgr2=0x7f97240b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:31.688 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.699+0000 7f9729ed3640 1 -- 172.21.15.57:0/878190617 shutdown_connections 2024-03-29T10:45:31.688 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.699+0000 7f9729ed3640 1 -- 172.21.15.57:0/878190617 wait complete. 2024-03-29T10:45:31.688 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.700+0000 7f9729ed3640 10 monclient: build_initial_monmap 2024-03-29T10:45:31.688 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.700+0000 7f9729ed3640 10 monclient: monmap: 2024-03-29T10:45:31.688 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:45:31.688 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:31.688 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:45:31.688 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:45:31.688 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:31.688 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:31.688 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:45:31.688 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:45:31.689 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:45:31.689 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:31.689 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.701+0000 7f9729ed3640 1 librados: starting msgr at 2024-03-29T10:45:31.689 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.701+0000 7f9729ed3640 1 librados: starting objecter 2024-03-29T10:45:31.689 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.701+0000 7f9729ed3640 1 Processor -- start 2024-03-29T10:45:31.689 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.703+0000 7f9729ed3640 1 -- start start 2024-03-29T10:45:31.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.703+0000 7f9729ed3640 1 librados: setting wanted keys 2024-03-29T10:45:31.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.703+0000 7f9729ed3640 1 librados: calling monclient init 2024-03-29T10:45:31.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.703+0000 7f9729ed3640 10 monclient: init 2024-03-29T10:45:31.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.704+0000 7f9729ed3640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:31.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.704+0000 7f9729ed3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:45:31.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.704+0000 7f9729ed3640 10 monclient(hunting): picked mon.noname-b con 0x7f97240ae250 addr v1:172.21.15.179:6789/0 2024-03-29T10:45:31.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.704+0000 7f9729ed3640 10 monclient(hunting): picked mon.noname-a con 0x7f972413f970 addr v1:172.21.15.57:6789/0 2024-03-29T10:45:31.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.704+0000 7f9729ed3640 10 monclient(hunting): picked mon.noname-c con 0x7f9724142800 addr v1:172.21.15.57:6790/0 2024-03-29T10:45:31.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.704+0000 7f9729ed3640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9724144a20 con 0x7f972413f970 2024-03-29T10:45:31.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.704+0000 7f9729ed3640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9724144bb0 con 0x7f97240ae250 2024-03-29T10:45:31.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.704+0000 7f9729ed3640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9724143590 con 0x7f9724142800 2024-03-29T10:45:31.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.704+0000 7f9729ed3640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:31.692 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.704+0000 7f9729ed3640 10 monclient(hunting): authenticate will time out at 1283.881200s 2024-03-29T10:45:31.694 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.707+0000 7f97237fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f97240ae250 0x7f972413a260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:59173/0 (socket says 172.21.15.57:59173) 2024-03-29T10:45:31.694 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.707+0000 7f97237fe640 1 -- 172.21.15.57:0/3330504053 learned_addr learned my addr 172.21.15.57:0/3330504053 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:45:31.694 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.707+0000 7f970bfff640 10 client.?.objecter ms_handle_connect 0x7f97240ae250 2024-03-29T10:45:31.701 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.707+0000 7f970bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:31.701 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.708+0000 7f970bfff640 1 -- 172.21.15.57:0/3330504053 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581809938 0 0) 0x7f9724144bb0 con 0x7f97240ae250 2024-03-29T10:45:31.701 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.708+0000 7f970bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:31.701 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.708+0000 7f970bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:31.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.708+0000 7f970bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:31.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.708+0000 7f970bfff640 10 monclient(hunting): my global_id is 4426 2024-03-29T10:45:31.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.708+0000 7f970bfff640 1 -- 172.21.15.57:0/3330504053 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9710004eb0 con 0x7f97240ae250 2024-03-29T10:45:31.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.709+0000 7f970bfff640 10 client.?.objecter ms_handle_connect 0x7f9724142800 2024-03-29T10:45:31.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.709+0000 7f970bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:31.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.709+0000 7f970bfff640 1 -- 172.21.15.57:0/3330504053 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439779706 0 0) 0x7f9724143590 con 0x7f9724142800 2024-03-29T10:45:31.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.709+0000 7f970bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:31.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.709+0000 7f970bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:31.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.709+0000 7f970bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:31.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.709+0000 7f970bfff640 10 monclient(hunting): my global_id is 4460 2024-03-29T10:45:31.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.709+0000 7f970bfff640 1 -- 172.21.15.57:0/3330504053 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9724144bb0 con 0x7f9724142800 2024-03-29T10:45:31.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.709+0000 7f970bfff640 10 client.?.objecter ms_handle_connect 0x7f972413f970 2024-03-29T10:45:31.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.709+0000 7f970bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:31.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.709+0000 7f970bfff640 1 -- 172.21.15.57:0/3330504053 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2241761607 0 0) 0x7f9724144bb0 con 0x7f9724142800 2024-03-29T10:45:31.703 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.709+0000 7f970bfff640 1 -- 172.21.15.57:0/3330504053 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9724143590 con 0x7f9724142800 2024-03-29T10:45:31.703 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.709+0000 7f970bfff640 1 -- 172.21.15.57:0/3330504053 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 55837420 0 0) 0x7f9724144a20 con 0x7f972413f970 2024-03-29T10:45:31.703 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.710+0000 7f970bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:31.703 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.710+0000 7f970bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:31.703 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.710+0000 7f970bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:31.703 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.710+0000 7f970bfff640 10 monclient(hunting): my global_id is 4416 2024-03-29T10:45:31.703 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.710+0000 7f970bfff640 1 -- 172.21.15.57:0/3330504053 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9724144bb0 con 0x7f972413f970 2024-03-29T10:45:31.703 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.710+0000 7f970bfff640 1 -- 172.21.15.57:0/3330504053 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f9714001750 con 0x7f9724142800 2024-03-29T10:45:31.703 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.710+0000 7f970bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:31.703 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.710+0000 7f970bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:31.703 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.710+0000 7f970bfff640 10 monclient(hunting): dump: 2024-03-29T10:45:31.703 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:31.704 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:31.704 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:31.704 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:31.704 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:31.704 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:31.704 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:31.704 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:31.704 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:31.704 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:31.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.710+0000 7f970bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:31.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.710+0000 7f970bfff640 1 -- 172.21.15.57:0/3330504053 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 258158947 0 0) 0x7f9710004eb0 con 0x7f97240ae250 2024-03-29T10:45:31.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.710+0000 7f970bfff640 1 -- 172.21.15.57:0/3330504053 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9724144a20 con 0x7f97240ae250 2024-03-29T10:45:31.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.710+0000 7f970bfff640 1 -- 172.21.15.57:0/3330504053 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2784025677 0 0) 0x7f9724143590 con 0x7f9724142800 2024-03-29T10:45:31.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.710+0000 7f970bfff640 1 -- 172.21.15.57:0/3330504053 >> v1:172.21.15.179:6789/0 conn(0x7f97240ae250 legacy=0x7f972413a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:31.705 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.710+0000 7f970bfff640 1 -- 172.21.15.57:0/3330504053 >> v1:172.21.15.57:6789/0 conn(0x7f972413f970 legacy=0x7f9724144310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:31.705 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.710+0000 7f970bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:45:31.705 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.710+0000 7f970bfff640 1 monclient: found mon.c 2024-03-29T10:45:31.705 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.710+0000 7f970bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:31.705 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.710+0000 7f970bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:45:31.705 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.710+0000 7f970bfff640 1 -- 172.21.15.57:0/3330504053 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9724143720 con 0x7f9724142800 2024-03-29T10:45:31.705 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.710+0000 7f970bfff640 10 monclient: _finish_auth 0 2024-03-29T10:45:31.705 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.710+0000 7f970bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:45:31.705 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.710+0000 7f970bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:31.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.711+0000 7f970bfff640 1 -- 172.21.15.57:0/3330504053 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97140018b0 con 0x7f9724142800 2024-03-29T10:45:31.708 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.711+0000 7f970bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:31.708 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.711+0000 7f970bfff640 1 -- 172.21.15.57:0/3330504053 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f9714001d60 con 0x7f9724142800 2024-03-29T10:45:31.708 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.711+0000 7f970bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:31.708 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.711+0000 7f970bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:31.708 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.711+0000 7f970bfff640 10 monclient: dump: 2024-03-29T10:45:31.708 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:31.708 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:31.708 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:31.708 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:31.708 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:31.708 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:31.708 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:31.708 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:31.709 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:31.709 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:31.709 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.712+0000 7f97217fa640 4 set_mon_vals no callback set 2024-03-29T10:45:31.709 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.713+0000 7f9729ed3640 5 monclient: authenticate success, global_id 4460 2024-03-29T10:45:31.709 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.713+0000 7f9729ed3640 10 monclient: _renew_subs 2024-03-29T10:45:31.709 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.713+0000 7f9729ed3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:45:31.709 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.713+0000 7f9729ed3640 1 -- 172.21.15.57:0/3330504053 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9724138220 con 0x7f9724142800 2024-03-29T10:45:31.709 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.713+0000 7f9729ed3640 10 client.4460.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:31.709 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.713+0000 7f9729ed3640 10 monclient: _renew_subs 2024-03-29T10:45:31.709 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.713+0000 7f9729ed3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:45:31.709 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.713+0000 7f9729ed3640 1 -- 172.21.15.57:0/3330504053 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f97240721d0 con 0x7f9724142800 2024-03-29T10:45:31.709 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.713+0000 7f9729ed3640 1 librados: init done 2024-03-29T10:45:31.709 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.714+0000 7f9729ed3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:31.709 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.714+0000 7f9729ed3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:31.710 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.715+0000 7f9729ed3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:45:31.710 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.715+0000 7f9729ed3640 1 -- 172.21.15.57:0/3330504053 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9724145ea0 con 0x7f9724142800 2024-03-29T10:45:31.710 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.718+0000 7f970bfff640 1 -- 172.21.15.57:0/3330504053 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f9714019600 con 0x7f9724142800 2024-03-29T10:45:31.710 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.719+0000 7f97237fe640 10 monclient: get_auth_request con 0x7f971006dfb0 auth_method 0 2024-03-29T10:45:31.710 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.719+0000 7f970bfff640 1 -- 172.21.15.57:0/3330504053 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f9714083670 con 0x7f9724142800 2024-03-29T10:45:31.710 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.719+0000 7f970bfff640 10 client.4460.objecter ms_dispatch 0x7f9724000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:31.710 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.719+0000 7f970bfff640 3 client.4460.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:31.710 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.719+0000 7f970bfff640 3 client.4460.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:31.710 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.719+0000 7f970bfff640 20 client.4460.objecter dump_active .. 0 homeless 2024-03-29T10:45:31.710 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.720+0000 7f970bfff640 10 client.4460.objecter ms_handle_connect 0x7f971006dfb0 2024-03-29T10:45:31.711 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.727+0000 7f970bfff640 1 -- 172.21.15.57:0/3330504053 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9724145ea0 con 0x7f9724142800 2024-03-29T10:45:31.712 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.727+0000 7f970bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:31.712 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:31.727+0000 7f970bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:32.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:32.061+0000 7f9729ed3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:32.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:32.061+0000 7f9729ed3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:32.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:32.061+0000 7f9729ed3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:45:32.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:32.061+0000 7f9729ed3640 1 -- 172.21.15.57:0/3330504053 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97240ae630 con 0x7f9724142800 2024-03-29T10:45:32.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:32.064+0000 7f970bfff640 1 -- 172.21.15.57:0/3330504053 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f97240ae630 con 0x7f9724142800 2024-03-29T10:45:32.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:32.064+0000 7f970bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:32.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:32.064+0000 7f970bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:32.048 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:45:32.048 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:32.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:32.067+0000 7f9729ed3640 10 librados: watch_flush enter 2024-03-29T10:45:32.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:32.068+0000 7f9729ed3640 10 librados: watch_flush exit 2024-03-29T10:45:32.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:32.068+0000 7f9729ed3640 20 client.4460.objecter shutdown clearing up homeless session... 2024-03-29T10:45:32.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:32.068+0000 7f9729ed3640 10 client.4460.objecter successfully canceled tick 2024-03-29T10:45:32.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:32.068+0000 7f9729ed3640 1 -- 172.21.15.57:0/3330504053 >> v1:172.21.15.179:6816/2698295936 conn(0x7f971006dfb0 legacy=0x7f9710070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:32.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:32.068+0000 7f9729ed3640 10 monclient: shutdown 2024-03-29T10:45:32.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:32.068+0000 7f9729ed3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:32.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:32.068+0000 7f9729ed3640 1 -- 172.21.15.57:0/3330504053 >> v1:172.21.15.57:6790/0 conn(0x7f9724142800 legacy=0x7f9724137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:32.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:32.068+0000 7f9729ed3640 1 -- 172.21.15.57:0/3330504053 shutdown_connections 2024-03-29T10:45:32.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:32.068+0000 7f9729ed3640 1 -- 172.21.15.57:0/3330504053 >> 172.21.15.57:0/3330504053 conn(0x7f97240b35e0 msgr2=0x7f97240a9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:32.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:32.069+0000 7f9729ed3640 1 -- 172.21.15.57:0/3330504053 shutdown_connections 2024-03-29T10:45:32.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:32.070+0000 7f9729ed3640 1 -- 172.21.15.57:0/3330504053 wait complete. 2024-03-29T10:45:32.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:32.070+0000 7f9729ed3640 1 librados: shutdown 2024-03-29T10:45:32.071 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:38.879 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:39.035 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.051+0000 7fdd23c25640 10 monclient: get_monmap_and_config 2024-03-29T10:45:39.035 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.051+0000 7fdd23c25640 10 monclient: build_initial_monmap 2024-03-29T10:45:39.035 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.051+0000 7fdd23c25640 10 monclient: monmap: 2024-03-29T10:45:39.035 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:45:39.035 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:39.035 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:45:39.053420+0000 2024-03-29T10:45:39.036 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:45:39.053420+0000 2024-03-29T10:45:39.036 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:39.036 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:39.036 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:45:39.036 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:45:39.036 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:45:39.036 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:39.037 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.053+0000 7fdd23c25640 1 Processor -- start 2024-03-29T10:45:39.039 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.054+0000 7fdd23c25640 1 -- start start 2024-03-29T10:45:39.039 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.054+0000 7fdd23c25640 10 monclient: init 2024-03-29T10:45:39.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.055+0000 7fdd23c25640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:39.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.055+0000 7fdd23c25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:45:39.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.055+0000 7fdd23c25640 10 monclient(hunting): picked mon.noname-a con 0x7fdd1c1457f0 addr v1:172.21.15.57:6789/0 2024-03-29T10:45:39.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.055+0000 7fdd23c25640 10 monclient(hunting): picked mon.noname-b con 0x7fdd1c13f650 addr v1:172.21.15.57:6790/0 2024-03-29T10:45:39.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.055+0000 7fdd23c25640 10 monclient(hunting): picked mon.noname-c con 0x7fdd1c0a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T10:45:39.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.055+0000 7fdd23c25640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1c0aaef0 con 0x7fdd1c1457f0 2024-03-29T10:45:39.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.055+0000 7fdd23c25640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1c145bd0 con 0x7fdd1c0a9d70 2024-03-29T10:45:39.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.055+0000 7fdd23c25640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1c145eb0 con 0x7fdd1c13f650 2024-03-29T10:45:39.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.055+0000 7fdd23c25640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:39.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.055+0000 7fdd23c25640 10 monclient(hunting): authenticate will time out at 1291.232357s 2024-03-29T10:45:39.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.059+0000 7fdd219c1640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fdd1c1457f0 0x7fdd1c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:38157/0 (socket says 172.21.15.57:38157) 2024-03-29T10:45:39.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.059+0000 7fdd219c1640 1 -- 172.21.15.57:0/2369336529 learned_addr learned my addr 172.21.15.57:0/2369336529 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:45:39.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.061+0000 7fdd209bf640 1 -- 172.21.15.57:0/2369336529 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479906343 0 0) 0x7fdd1c145eb0 con 0x7fdd1c13f650 2024-03-29T10:45:39.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.061+0000 7fdd209bf640 10 monclient(hunting): _negotiate 2024-03-29T10:45:39.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.061+0000 7fdd209bf640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:39.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.061+0000 7fdd209bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:39.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.061+0000 7fdd209bf640 10 monclient(hunting): my global_id is 4463 2024-03-29T10:45:39.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd209bf640 1 -- 172.21.15.57:0/2369336529 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd08003520 con 0x7fdd1c13f650 2024-03-29T10:45:39.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd209bf640 1 -- 172.21.15.57:0/2369336529 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011184272 0 0) 0x7fdd1c0aaef0 con 0x7fdd1c1457f0 2024-03-29T10:45:39.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd209bf640 10 monclient(hunting): _negotiate 2024-03-29T10:45:39.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd209bf640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:39.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd209bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:39.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd209bf640 10 monclient(hunting): my global_id is 4419 2024-03-29T10:45:39.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd209bf640 1 -- 172.21.15.57:0/2369336529 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd1c145eb0 con 0x7fdd1c1457f0 2024-03-29T10:45:39.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd209bf640 1 -- 172.21.15.57:0/2369336529 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3146949553 0 0) 0x7fdd1c145eb0 con 0x7fdd1c1457f0 2024-03-29T10:45:39.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd209bf640 1 -- 172.21.15.57:0/2369336529 >> v1:172.21.15.57:6790/0 conn(0x7fdd1c13f650 legacy=0x7fdd1c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:39.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd209bf640 1 -- 172.21.15.57:0/2369336529 >> v1:172.21.15.179:6789/0 conn(0x7fdd1c0a9d70 legacy=0x7fdd1c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:39.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd209bf640 10 monclient: _finish_hunting 0 2024-03-29T10:45:39.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd209bf640 1 monclient: found mon.noname-a 2024-03-29T10:45:39.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd209bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:39.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd209bf640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:45:39.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd209bf640 1 -- 172.21.15.57:0/2369336529 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1c146340 con 0x7fdd1c1457f0 2024-03-29T10:45:39.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd209bf640 10 monclient: _finish_auth 0 2024-03-29T10:45:39.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd209bf640 10 monclient: _check_auth_tickets 2024-03-29T10:45:39.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd209bf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:39.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd23c25640 5 monclient: authenticate success, global_id 4419 2024-03-29T10:45:39.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd23c25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:39.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd209bf640 1 -- 172.21.15.57:0/2369336529 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fdd180015f0 con 0x7fdd1c1457f0 2024-03-29T10:45:39.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd209bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:39.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd209bf640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:39.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd209bf640 10 monclient: dump: 2024-03-29T10:45:39.047 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:39.047 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:39.048 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:39.048 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:39.048 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:39.048 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:39.048 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:39.048 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:39.048 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:39.048 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:39.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd23c25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:39.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd209bf640 1 -- 172.21.15.57:0/2369336529 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd18001840 con 0x7fdd1c1457f0 2024-03-29T10:45:39.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd209bf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:39.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd23c25640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:39.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.062+0000 7fdd23c25640 4 set_mon_vals no callback set 2024-03-29T10:45:39.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.063+0000 7fdd23c25640 10 monclient: shutdown 2024-03-29T10:45:39.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.063+0000 7fdd23c25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:39.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.063+0000 7fdd23c25640 1 -- 172.21.15.57:0/2369336529 >> v1:172.21.15.57:6789/0 conn(0x7fdd1c1457f0 legacy=0x7fdd1c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:39.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.066+0000 7fdd23c25640 1 -- 172.21.15.57:0/2369336529 shutdown_connections 2024-03-29T10:45:39.050 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.066+0000 7fdd23c25640 1 -- 172.21.15.57:0/2369336529 >> 172.21.15.57:0/2369336529 conn(0x7fdd1c066a00 msgr2=0x7fdd1c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:39.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.066+0000 7fdd23c25640 1 -- 172.21.15.57:0/2369336529 shutdown_connections 2024-03-29T10:45:39.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.066+0000 7fdd23c25640 1 -- 172.21.15.57:0/2369336529 wait complete. 2024-03-29T10:45:39.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.067+0000 7fdd23c25640 10 monclient: build_initial_monmap 2024-03-29T10:45:39.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.067+0000 7fdd23c25640 10 monclient: monmap: 2024-03-29T10:45:39.052 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:45:39.052 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:39.052 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:45:39.053 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:45:39.053 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:39.053 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:39.053 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:45:39.053 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:45:39.053 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:45:39.053 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:39.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.068+0000 7fdd23c25640 1 librados: starting msgr at 2024-03-29T10:45:39.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.068+0000 7fdd23c25640 1 librados: starting objecter 2024-03-29T10:45:39.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.068+0000 7fdd23c25640 1 Processor -- start 2024-03-29T10:45:39.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.068+0000 7fdd23c25640 1 -- start start 2024-03-29T10:45:39.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.068+0000 7fdd23c25640 1 librados: setting wanted keys 2024-03-29T10:45:39.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.068+0000 7fdd23c25640 1 librados: calling monclient init 2024-03-29T10:45:39.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.068+0000 7fdd23c25640 10 monclient: init 2024-03-29T10:45:39.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.069+0000 7fdd23c25640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:39.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.069+0000 7fdd23c25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:45:39.054 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.069+0000 7fdd23c25640 10 monclient(hunting): picked mon.noname-a con 0x7fdd1c0a9d70 addr v1:172.21.15.57:6789/0 2024-03-29T10:45:39.054 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.069+0000 7fdd23c25640 10 monclient(hunting): picked mon.noname-c con 0x7fdd1c13f650 addr v1:172.21.15.57:6790/0 2024-03-29T10:45:39.056 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.072+0000 7fdd23c25640 10 monclient(hunting): picked mon.noname-b con 0x7fdd1c1457f0 addr v1:172.21.15.179:6789/0 2024-03-29T10:45:39.056 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.072+0000 7fdd23c25640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1c142ba0 con 0x7fdd1c0a9d70 2024-03-29T10:45:39.056 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.072+0000 7fdd23c25640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1c06eb40 con 0x7fdd1c1457f0 2024-03-29T10:45:39.056 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.072+0000 7fdd23c25640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1c142780 con 0x7fdd1c13f650 2024-03-29T10:45:39.056 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.072+0000 7fdd23c25640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:39.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.072+0000 7fdd23c25640 10 monclient(hunting): authenticate will time out at 1291.249479s 2024-03-29T10:45:39.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.072+0000 7fdd219c1640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fdd1c0a9d70 0x7fdd1c142070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:38413/0 (socket says 172.21.15.57:38413) 2024-03-29T10:45:39.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.072+0000 7fdd219c1640 1 -- 172.21.15.57:0/2349925714 learned_addr learned my addr 172.21.15.57:0/2349925714 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:45:39.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.072+0000 7fdd127fc640 10 client.?.objecter ms_handle_connect 0x7fdd1c0a9d70 2024-03-29T10:45:39.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.072+0000 7fdd127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:39.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.073+0000 7fdd127fc640 1 -- 172.21.15.57:0/2349925714 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3005637305 0 0) 0x7fdd1c142ba0 con 0x7fdd1c0a9d70 2024-03-29T10:45:39.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.073+0000 7fdd127fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:39.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.073+0000 7fdd127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:39.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.073+0000 7fdd127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:39.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.073+0000 7fdd127fc640 10 monclient(hunting): my global_id is 4422 2024-03-29T10:45:39.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.073+0000 7fdd127fc640 1 -- 172.21.15.57:0/2349925714 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd00004eb0 con 0x7fdd1c0a9d70 2024-03-29T10:45:39.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.073+0000 7fdd127fc640 10 client.?.objecter ms_handle_connect 0x7fdd1c1457f0 2024-03-29T10:45:39.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.073+0000 7fdd127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:39.058 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.073+0000 7fdd127fc640 1 -- 172.21.15.57:0/2349925714 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1148162753 0 0) 0x7fdd00004eb0 con 0x7fdd1c0a9d70 2024-03-29T10:45:39.058 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.073+0000 7fdd127fc640 1 -- 172.21.15.57:0/2349925714 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd00005300 con 0x7fdd1c0a9d70 2024-03-29T10:45:39.058 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.073+0000 7fdd127fc640 1 -- 172.21.15.57:0/2349925714 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fdd18004980 con 0x7fdd1c0a9d70 2024-03-29T10:45:39.058 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.073+0000 7fdd127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:39.058 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.073+0000 7fdd127fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:39.058 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.073+0000 7fdd127fc640 10 monclient(hunting): dump: 2024-03-29T10:45:39.058 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:39.058 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:39.058 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:39.058 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:39.058 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:39.059 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:39.059 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:39.059 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:39.059 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:39.059 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:39.059 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.073+0000 7fdd127fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:39.059 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.074+0000 7fdd127fc640 1 -- 172.21.15.57:0/2349925714 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1806524950 0 0) 0x7fdd00005300 con 0x7fdd1c0a9d70 2024-03-29T10:45:39.059 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.074+0000 7fdd127fc640 1 -- 172.21.15.57:0/2349925714 >> v1:172.21.15.57:6790/0 conn(0x7fdd1c13f650 legacy=0x7fdd1c136e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:39.060 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.075+0000 7fdd127fc640 1 -- 172.21.15.57:0/2349925714 >> v1:172.21.15.179:6789/0 conn(0x7fdd1c1457f0 legacy=0x7fdd1c1d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:39.060 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.075+0000 7fdd127fc640 10 monclient: _finish_hunting 0 2024-03-29T10:45:39.060 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.075+0000 7fdd127fc640 1 monclient: found mon.a 2024-03-29T10:45:39.061 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.075+0000 7fdd127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:39.061 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.075+0000 7fdd127fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:45:39.061 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.075+0000 7fdd127fc640 1 -- 172.21.15.57:0/2349925714 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1c137540 con 0x7fdd1c0a9d70 2024-03-29T10:45:39.061 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.075+0000 7fdd127fc640 10 monclient: _finish_auth 0 2024-03-29T10:45:39.061 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.075+0000 7fdd127fc640 10 monclient: _check_auth_tickets 2024-03-29T10:45:39.061 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.075+0000 7fdd127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:39.061 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.075+0000 7fdd23c25640 5 monclient: authenticate success, global_id 4422 2024-03-29T10:45:39.061 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.075+0000 7fdd23c25640 10 monclient: _renew_subs 2024-03-29T10:45:39.061 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.075+0000 7fdd23c25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:45:39.061 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.075+0000 7fdd23c25640 1 -- 172.21.15.57:0/2349925714 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd1c137790 con 0x7fdd1c0a9d70 2024-03-29T10:45:39.061 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.075+0000 7fdd23c25640 10 client.4422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:39.062 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.075+0000 7fdd23c25640 10 monclient: _renew_subs 2024-03-29T10:45:39.062 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.075+0000 7fdd23c25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:45:39.062 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.075+0000 7fdd23c25640 1 -- 172.21.15.57:0/2349925714 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd1c144a20 con 0x7fdd1c0a9d70 2024-03-29T10:45:39.062 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.075+0000 7fdd23c25640 1 librados: init done 2024-03-29T10:45:39.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.077+0000 7fdd127fc640 1 -- 172.21.15.57:0/2349925714 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd18001a30 con 0x7fdd1c0a9d70 2024-03-29T10:45:39.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.077+0000 7fdd127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:39.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.077+0000 7fdd13fff640 4 set_mon_vals no callback set 2024-03-29T10:45:39.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.078+0000 7fdd23c25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:39.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.078+0000 7fdd127fc640 1 -- 172.21.15.57:0/2349925714 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fdd18004980 con 0x7fdd1c0a9d70 2024-03-29T10:45:39.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.078+0000 7fdd23c25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:39.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.078+0000 7fdd23c25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:45:39.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.078+0000 7fdd23c25640 1 -- 172.21.15.57:0/2349925714 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd04005080 con 0x7fdd1c0a9d70 2024-03-29T10:45:39.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.078+0000 7fdd127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:39.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.078+0000 7fdd127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:39.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.078+0000 7fdd127fc640 10 monclient: dump: 2024-03-29T10:45:39.064 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:39.064 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:39.064 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:39.065 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:39.065 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:39.065 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:39.065 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:39.065 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:39.065 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:39.065 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:39.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.079+0000 7fdd127fc640 1 -- 172.21.15.57:0/2349925714 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fdd18004e10 con 0x7fdd1c0a9d70 2024-03-29T10:45:39.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.079+0000 7fdd127fc640 1 -- 172.21.15.57:0/2349925714 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7fdd18084a40 con 0x7fdd1c0a9d70 2024-03-29T10:45:39.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.079+0000 7fdd127fc640 10 client.4422.objecter ms_dispatch 0x7fdd1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:39.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.079+0000 7fdd127fc640 3 client.4422.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:39.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.079+0000 7fdd127fc640 3 client.4422.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:39.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.080+0000 7fdd127fc640 20 client.4422.objecter dump_active .. 0 homeless 2024-03-29T10:45:39.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.080+0000 7fdd211c0640 10 monclient: get_auth_request con 0x7fdd0006e050 auth_method 0 2024-03-29T10:45:39.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.080+0000 7fdd127fc640 10 client.4422.objecter ms_handle_connect 0x7fdd0006e050 2024-03-29T10:45:39.072 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.088+0000 7fdd127fc640 1 -- 172.21.15.57:0/2349925714 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdd18050320 con 0x7fdd1c0a9d70 2024-03-29T10:45:39.072 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.088+0000 7fdd127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:39.072 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.088+0000 7fdd127fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:39.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.425+0000 7fdd23c25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:39.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.425+0000 7fdd23c25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:39.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.425+0000 7fdd23c25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:45:39.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.425+0000 7fdd23c25640 1 -- 172.21.15.57:0/2349925714 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd04005330 con 0x7fdd1c0a9d70 2024-03-29T10:45:39.411 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.428+0000 7fdd127fc640 1 -- 172.21.15.57:0/2349925714 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7fdd18050320 con 0x7fdd1c0a9d70 2024-03-29T10:45:39.411 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.428+0000 7fdd127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:39.411 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.428+0000 7fdd127fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:39.415 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:45:39.415 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:39.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.434+0000 7fdd23c25640 10 librados: watch_flush enter 2024-03-29T10:45:39.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.434+0000 7fdd23c25640 10 librados: watch_flush exit 2024-03-29T10:45:39.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.434+0000 7fdd23c25640 20 client.4422.objecter shutdown clearing up homeless session... 2024-03-29T10:45:39.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.434+0000 7fdd23c25640 10 client.4422.objecter successfully canceled tick 2024-03-29T10:45:39.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.434+0000 7fdd23c25640 1 -- 172.21.15.57:0/2349925714 >> v1:172.21.15.179:6816/2698295936 conn(0x7fdd0006e050 legacy=0x7fdd00070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:39.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.434+0000 7fdd23c25640 10 monclient: shutdown 2024-03-29T10:45:39.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.434+0000 7fdd23c25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:39.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.434+0000 7fdd23c25640 1 -- 172.21.15.57:0/2349925714 >> v1:172.21.15.57:6789/0 conn(0x7fdd1c0a9d70 legacy=0x7fdd1c142070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:39.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.434+0000 7fdd23c25640 1 -- 172.21.15.57:0/2349925714 shutdown_connections 2024-03-29T10:45:39.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.434+0000 7fdd23c25640 1 -- 172.21.15.57:0/2349925714 >> 172.21.15.57:0/2349925714 conn(0x7fdd1c066a00 msgr2=0x7fdd1c0b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:39.419 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.435+0000 7fdd23c25640 1 -- 172.21.15.57:0/2349925714 shutdown_connections 2024-03-29T10:45:39.419 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.435+0000 7fdd23c25640 1 -- 172.21.15.57:0/2349925714 wait complete. 2024-03-29T10:45:39.419 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:39.435+0000 7fdd23c25640 1 librados: shutdown 2024-03-29T10:45:39.435 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:46.243 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:46.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.410+0000 7f5dbb864640 10 monclient: get_monmap_and_config 2024-03-29T10:45:46.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.410+0000 7f5dbb864640 10 monclient: build_initial_monmap 2024-03-29T10:45:46.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.410+0000 7f5dbb864640 10 monclient: monmap: 2024-03-29T10:45:46.394 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:45:46.394 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:46.394 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:45:46.411924+0000 2024-03-29T10:45:46.394 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:45:46.411924+0000 2024-03-29T10:45:46.394 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:46.394 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:46.394 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:45:46.394 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:45:46.395 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:45:46.395 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:46.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.411+0000 7f5dbb864640 1 Processor -- start 2024-03-29T10:45:46.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.411+0000 7f5dbb864640 1 -- start start 2024-03-29T10:45:46.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.411+0000 7f5dbb864640 10 monclient: init 2024-03-29T10:45:46.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.412+0000 7f5dbb864640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:46.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.412+0000 7f5dbb864640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:45:46.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.412+0000 7f5dbb864640 10 monclient(hunting): picked mon.noname-b con 0x7f5db41457f0 addr v1:172.21.15.57:6790/0 2024-03-29T10:45:46.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.412+0000 7f5dbb864640 10 monclient(hunting): picked mon.noname-c con 0x7f5db413f650 addr v1:172.21.15.179:6789/0 2024-03-29T10:45:46.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.412+0000 7f5dbb864640 10 monclient(hunting): picked mon.noname-a con 0x7f5db40a9d70 addr v1:172.21.15.57:6789/0 2024-03-29T10:45:46.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.412+0000 7f5dbb864640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db40aaef0 con 0x7f5db40a9d70 2024-03-29T10:45:46.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.412+0000 7f5dbb864640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db4145bd0 con 0x7f5db413f650 2024-03-29T10:45:46.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.412+0000 7f5dbb864640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db4145eb0 con 0x7f5db41457f0 2024-03-29T10:45:46.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.412+0000 7f5dbb864640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:46.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.412+0000 7f5dbb864640 10 monclient(hunting): authenticate will time out at 1298.589811s 2024-03-29T10:45:46.401 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.418+0000 7f5db9e01640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f5db40a9d70 0x7f5db414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46293/0 (socket says 172.21.15.57:46293) 2024-03-29T10:45:46.401 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.418+0000 7f5db9e01640 1 -- 172.21.15.57:0/3751874912 learned_addr learned my addr 172.21.15.57:0/3751874912 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:45:46.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.418+0000 7f5db3fff640 1 -- 172.21.15.57:0/3751874912 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 486332070 0 0) 0x7f5db40aaef0 con 0x7f5db40a9d70 2024-03-29T10:45:46.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.419+0000 7f5db3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:46.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.419+0000 7f5db3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:46.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.419+0000 7f5db3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:46.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.419+0000 7f5db3fff640 10 monclient(hunting): my global_id is 4425 2024-03-29T10:45:46.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.419+0000 7f5db3fff640 1 -- 172.21.15.57:0/3751874912 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d9c003520 con 0x7f5db40a9d70 2024-03-29T10:45:46.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.419+0000 7f5db3fff640 1 -- 172.21.15.57:0/3751874912 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4070948058 0 0) 0x7f5d9c003520 con 0x7f5db40a9d70 2024-03-29T10:45:46.403 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.419+0000 7f5db3fff640 1 -- 172.21.15.57:0/3751874912 >> v1:172.21.15.57:6790/0 conn(0x7f5db41457f0 legacy=0x7f5db413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:46.407 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.421+0000 7f5db3fff640 1 -- 172.21.15.57:0/3751874912 >> v1:172.21.15.179:6789/0 conn(0x7f5db413f650 legacy=0x7f5db40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:46.407 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.421+0000 7f5db3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:45:46.407 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.421+0000 7f5db3fff640 1 monclient: found mon.noname-a 2024-03-29T10:45:46.407 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.421+0000 7f5db3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:46.407 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.421+0000 7f5db3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:45:46.407 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.421+0000 7f5db3fff640 1 -- 172.21.15.57:0/3751874912 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db4146340 con 0x7f5db40a9d70 2024-03-29T10:45:46.407 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.421+0000 7f5db3fff640 10 monclient: _finish_auth 0 2024-03-29T10:45:46.407 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.421+0000 7f5db3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:45:46.407 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.421+0000 7f5db3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:46.408 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.421+0000 7f5dbb864640 5 monclient: authenticate success, global_id 4425 2024-03-29T10:45:46.408 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.421+0000 7f5dbb864640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:46.408 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.421+0000 7f5db3fff640 1 -- 172.21.15.57:0/3751874912 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f5da4001690 con 0x7f5db40a9d70 2024-03-29T10:45:46.408 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.421+0000 7f5db3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:46.408 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.421+0000 7f5db3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:46.408 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.421+0000 7f5db3fff640 10 monclient: dump: 2024-03-29T10:45:46.408 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:46.408 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:46.408 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:46.408 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:46.408 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:46.408 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:46.408 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:46.408 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:46.408 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:46.409 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:46.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.421+0000 7f5dbb864640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:46.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.424+0000 7f5db3fff640 1 -- 172.21.15.57:0/3751874912 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5da4001920 con 0x7f5db40a9d70 2024-03-29T10:45:46.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.424+0000 7f5db3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:46.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.424+0000 7f5dbb864640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:46.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.424+0000 7f5dbb864640 4 set_mon_vals no callback set 2024-03-29T10:45:46.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.425+0000 7f5dbb864640 10 monclient: shutdown 2024-03-29T10:45:46.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.425+0000 7f5dbb864640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:46.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.425+0000 7f5dbb864640 1 -- 172.21.15.57:0/3751874912 >> v1:172.21.15.57:6789/0 conn(0x7f5db40a9d70 legacy=0x7f5db414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:46.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.426+0000 7f5dbb864640 1 -- 172.21.15.57:0/3751874912 shutdown_connections 2024-03-29T10:45:46.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.426+0000 7f5dbb864640 1 -- 172.21.15.57:0/3751874912 >> 172.21.15.57:0/3751874912 conn(0x7f5db4066a00 msgr2=0x7f5db4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:46.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.428+0000 7f5dbb864640 1 -- 172.21.15.57:0/3751874912 shutdown_connections 2024-03-29T10:45:46.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.429+0000 7f5dbb864640 1 -- 172.21.15.57:0/3751874912 wait complete. 2024-03-29T10:45:46.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.430+0000 7f5dbb864640 10 monclient: build_initial_monmap 2024-03-29T10:45:46.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.430+0000 7f5dbb864640 10 monclient: monmap: 2024-03-29T10:45:46.415 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:45:46.415 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:46.415 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:45:46.415 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:45:46.415 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:46.415 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:46.415 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:45:46.415 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:45:46.415 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:45:46.415 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:46.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.430+0000 7f5dbb864640 1 librados: starting msgr at 2024-03-29T10:45:46.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.430+0000 7f5dbb864640 1 librados: starting objecter 2024-03-29T10:45:46.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.430+0000 7f5dbb864640 1 Processor -- start 2024-03-29T10:45:46.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.430+0000 7f5dbb864640 1 -- start start 2024-03-29T10:45:46.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.430+0000 7f5dbb864640 1 librados: setting wanted keys 2024-03-29T10:45:46.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.430+0000 7f5dbb864640 1 librados: calling monclient init 2024-03-29T10:45:46.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.430+0000 7f5dbb864640 10 monclient: init 2024-03-29T10:45:46.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.431+0000 7f5dbb864640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:46.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.431+0000 7f5dbb864640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:45:46.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.431+0000 7f5dbb864640 10 monclient(hunting): picked mon.noname-b con 0x7f5db40a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T10:45:46.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.431+0000 7f5dbb864640 10 monclient(hunting): picked mon.noname-a con 0x7f5db413f650 addr v1:172.21.15.57:6789/0 2024-03-29T10:45:46.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.431+0000 7f5dbb864640 10 monclient(hunting): picked mon.noname-c con 0x7f5db41457f0 addr v1:172.21.15.57:6790/0 2024-03-29T10:45:46.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.431+0000 7f5dbb864640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db41414c0 con 0x7f5db413f650 2024-03-29T10:45:46.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.431+0000 7f5dbb864640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db4137fb0 con 0x7f5db40a9d70 2024-03-29T10:45:46.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.431+0000 7f5dbb864640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db406efd0 con 0x7f5db41457f0 2024-03-29T10:45:46.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.431+0000 7f5dbb864640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:46.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.431+0000 7f5dbb864640 10 monclient(hunting): authenticate will time out at 1298.608567s 2024-03-29T10:45:46.419 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.436+0000 7f5db8dff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f5db413f650 0x7f5db4140db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46675/0 (socket says 172.21.15.57:46675) 2024-03-29T10:45:46.419 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.436+0000 7f5db8dff640 1 -- 172.21.15.57:0/1264993831 learned_addr learned my addr 172.21.15.57:0/1264993831 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:45:46.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.436+0000 7f5db1ffb640 10 client.?.objecter ms_handle_connect 0x7f5db413f650 2024-03-29T10:45:46.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.437+0000 7f5db1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:46.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.437+0000 7f5db1ffb640 1 -- 172.21.15.57:0/1264993831 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2624683185 0 0) 0x7f5db41414c0 con 0x7f5db413f650 2024-03-29T10:45:46.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.437+0000 7f5db1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:45:46.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.437+0000 7f5db1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:46.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.437+0000 7f5db1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:46.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.437+0000 7f5db1ffb640 10 monclient(hunting): my global_id is 4428 2024-03-29T10:45:46.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.437+0000 7f5db1ffb640 1 -- 172.21.15.57:0/1264993831 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5da0004eb0 con 0x7f5db413f650 2024-03-29T10:45:46.421 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.437+0000 7f5db1ffb640 1 -- 172.21.15.57:0/1264993831 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1810778686 0 0) 0x7f5da0004eb0 con 0x7f5db413f650 2024-03-29T10:45:46.421 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.437+0000 7f5db1ffb640 1 -- 172.21.15.57:0/1264993831 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5db41414c0 con 0x7f5db413f650 2024-03-29T10:45:46.421 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.437+0000 7f5db1ffb640 1 -- 172.21.15.57:0/1264993831 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f5dac001d90 con 0x7f5db413f650 2024-03-29T10:45:46.421 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.437+0000 7f5db1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:46.421 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.437+0000 7f5db1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:46.421 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.437+0000 7f5db1ffb640 10 monclient(hunting): dump: 2024-03-29T10:45:46.421 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:46.421 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:46.422 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:46.422 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:46.422 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:46.422 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:46.422 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:46.422 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:46.422 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:46.422 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:46.422 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.437+0000 7f5db1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:46.422 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.437+0000 7f5db1ffb640 1 -- 172.21.15.57:0/1264993831 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3579232162 0 0) 0x7f5db41414c0 con 0x7f5db413f650 2024-03-29T10:45:46.422 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.437+0000 7f5db1ffb640 1 -- 172.21.15.57:0/1264993831 >> v1:172.21.15.57:6790/0 conn(0x7f5db41457f0 legacy=0x7f5db41378a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:46.422 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.438+0000 7f5db1ffb640 1 -- 172.21.15.57:0/1264993831 >> v1:172.21.15.179:6789/0 conn(0x7f5db40a9d70 legacy=0x7f5db4137190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:46.423 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.440+0000 7f5db1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:45:46.423 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.440+0000 7f5db1ffb640 1 monclient: found mon.a 2024-03-29T10:45:46.423 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.440+0000 7f5db1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:46.423 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.440+0000 7f5db1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:45:46.423 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.440+0000 7f5db1ffb640 1 -- 172.21.15.57:0/1264993831 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db406f2b0 con 0x7f5db413f650 2024-03-29T10:45:46.424 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.440+0000 7f5db1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:45:46.424 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.440+0000 7f5db1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:45:46.424 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.440+0000 7f5db1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:46.424 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.440+0000 7f5dbb864640 5 monclient: authenticate success, global_id 4428 2024-03-29T10:45:46.424 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.440+0000 7f5dbb864640 10 monclient: _renew_subs 2024-03-29T10:45:46.424 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.440+0000 7f5dbb864640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:45:46.424 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.440+0000 7f5dbb864640 1 -- 172.21.15.57:0/1264993831 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5db406f590 con 0x7f5db413f650 2024-03-29T10:45:46.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.440+0000 7f5dbb864640 10 client.4428.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:46.429 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.440+0000 7f5dbb864640 10 monclient: _renew_subs 2024-03-29T10:45:46.429 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.440+0000 7f5dbb864640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:45:46.429 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.440+0000 7f5dbb864640 1 -- 172.21.15.57:0/1264993831 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5db406fd70 con 0x7f5db413f650 2024-03-29T10:45:46.429 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.440+0000 7f5dbb864640 1 librados: init done 2024-03-29T10:45:46.429 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.441+0000 7f5dbb864640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:46.429 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.441+0000 7f5dbb864640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:46.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.441+0000 7f5dbb864640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:45:46.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.441+0000 7f5dbb864640 1 -- 172.21.15.57:0/1264993831 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5db406fab0 con 0x7f5db413f650 2024-03-29T10:45:46.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.443+0000 7f5db1ffb640 1 -- 172.21.15.57:0/1264993831 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5dac001ef0 con 0x7f5db413f650 2024-03-29T10:45:46.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.443+0000 7f5db1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:46.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.443+0000 7f5db1ffb640 1 -- 172.21.15.57:0/1264993831 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f5dac003de0 con 0x7f5db413f650 2024-03-29T10:45:46.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.443+0000 7f5db1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:46.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.443+0000 7f5db1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:46.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.443+0000 7f5db1ffb640 10 monclient: dump: 2024-03-29T10:45:46.430 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:46.430 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:46.430 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:46.430 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:46.430 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:46.430 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:46.431 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:46.431 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:46.431 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:46.431 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:46.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.443+0000 7f5db1ffb640 1 -- 172.21.15.57:0/1264993831 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f5dac004000 con 0x7f5db413f650 2024-03-29T10:45:46.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.443+0000 7f5db37fe640 4 set_mon_vals no callback set 2024-03-29T10:45:46.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.445+0000 7f5db9600640 10 monclient: get_auth_request con 0x7f5da006dce0 auth_method 0 2024-03-29T10:45:46.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.448+0000 7f5db1ffb640 10 client.4428.objecter ms_handle_connect 0x7f5da006dce0 2024-03-29T10:45:46.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.448+0000 7f5db1ffb640 1 -- 172.21.15.57:0/1264993831 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f5dac054ee0 con 0x7f5db413f650 2024-03-29T10:45:46.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.448+0000 7f5db1ffb640 10 client.4428.objecter ms_dispatch 0x7f5db4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:46.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.448+0000 7f5db1ffb640 3 client.4428.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:46.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.448+0000 7f5db1ffb640 3 client.4428.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:46.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.448+0000 7f5db1ffb640 20 client.4428.objecter dump_active .. 0 homeless 2024-03-29T10:45:46.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.452+0000 7f5db1ffb640 1 -- 172.21.15.57:0/1264993831 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5db406fab0 con 0x7f5db413f650 2024-03-29T10:45:46.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.452+0000 7f5db1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:46.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.452+0000 7f5db1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:46.772 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.788+0000 7f5dbb864640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:46.772 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.788+0000 7f5dbb864640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:46.772 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.788+0000 7f5dbb864640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:45:46.772 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.788+0000 7f5dbb864640 1 -- 172.21.15.57:0/1264993831 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5db406fab0 con 0x7f5db413f650 2024-03-29T10:45:46.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.789+0000 7f5db1ffb640 1 -- 172.21.15.57:0/1264993831 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f5db406fab0 con 0x7f5db413f650 2024-03-29T10:45:46.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.789+0000 7f5db1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:46.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.789+0000 7f5db1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:46.776 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:45:46.776 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:46.779 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.796+0000 7f5dbb864640 10 librados: watch_flush enter 2024-03-29T10:45:46.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.797+0000 7f5dbb864640 10 librados: watch_flush exit 2024-03-29T10:45:46.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.797+0000 7f5dbb864640 20 client.4428.objecter shutdown clearing up homeless session... 2024-03-29T10:45:46.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.797+0000 7f5dbb864640 10 client.4428.objecter successfully canceled tick 2024-03-29T10:45:46.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.797+0000 7f5dbb864640 1 -- 172.21.15.57:0/1264993831 >> v1:172.21.15.179:6816/2698295936 conn(0x7f5da006dce0 legacy=0x7f5da00701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:46.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.797+0000 7f5dbb864640 10 monclient: shutdown 2024-03-29T10:45:46.781 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.797+0000 7f5dbb864640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:46.782 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.797+0000 7f5dbb864640 1 -- 172.21.15.57:0/1264993831 >> v1:172.21.15.57:6789/0 conn(0x7f5db413f650 legacy=0x7f5db4140db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:46.782 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.797+0000 7f5dbb864640 1 -- 172.21.15.57:0/1264993831 shutdown_connections 2024-03-29T10:45:46.782 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.797+0000 7f5dbb864640 1 -- 172.21.15.57:0/1264993831 >> 172.21.15.57:0/1264993831 conn(0x7f5db4066a00 msgr2=0x7f5db40b4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:46.782 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.797+0000 7f5dbb864640 1 -- 172.21.15.57:0/1264993831 shutdown_connections 2024-03-29T10:45:46.782 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.797+0000 7f5dbb864640 1 -- 172.21.15.57:0/1264993831 wait complete. 2024-03-29T10:45:46.782 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:46.797+0000 7f5dbb864640 1 librados: shutdown 2024-03-29T10:45:46.799 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:53.607 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:53.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.773+0000 7f2f8020d640 10 monclient: get_monmap_and_config 2024-03-29T10:45:53.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.773+0000 7f2f8020d640 10 monclient: build_initial_monmap 2024-03-29T10:45:53.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.773+0000 7f2f8020d640 10 monclient: monmap: 2024-03-29T10:45:53.756 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:45:53.756 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:53.756 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:45:53.774069+0000 2024-03-29T10:45:53.756 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:45:53.774069+0000 2024-03-29T10:45:53.756 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:53.756 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:53.756 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:45:53.756 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:45:53.756 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:45:53.756 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:53.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.774+0000 7f2f8020d640 1 Processor -- start 2024-03-29T10:45:53.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.774+0000 7f2f8020d640 1 -- start start 2024-03-29T10:45:53.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.774+0000 7f2f8020d640 10 monclient: init 2024-03-29T10:45:53.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.775+0000 7f2f8020d640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:53.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.775+0000 7f2f8020d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:45:53.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.775+0000 7f2f8020d640 10 monclient(hunting): picked mon.noname-a con 0x7f2f780b0770 addr v1:172.21.15.57:6789/0 2024-03-29T10:45:53.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.775+0000 7f2f8020d640 10 monclient(hunting): picked mon.noname-b con 0x7f2f7814a700 addr v1:172.21.15.57:6790/0 2024-03-29T10:45:53.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.775+0000 7f2f8020d640 10 monclient(hunting): picked mon.noname-c con 0x7f2f780ac5a0 addr v1:172.21.15.179:6789/0 2024-03-29T10:45:53.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.775+0000 7f2f8020d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f780aba70 con 0x7f2f780b0770 2024-03-29T10:45:53.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.775+0000 7f2f8020d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f780aecb0 con 0x7f2f780ac5a0 2024-03-29T10:45:53.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.775+0000 7f2f8020d640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f780af150 con 0x7f2f7814a700 2024-03-29T10:45:53.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.775+0000 7f2f8020d640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:53.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.775+0000 7f2f8020d640 10 monclient(hunting): authenticate will time out at 1305.951599s 2024-03-29T10:45:53.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.775+0000 7f2f7d7a8640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f2f7814a700 0x7f2f780abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:39456/0 (socket says 172.21.15.57:39456) 2024-03-29T10:45:53.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.776+0000 7f2f7d7a8640 1 -- 172.21.15.57:0/4136826866 learned_addr learned my addr 172.21.15.57:0/4136826866 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:45:53.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.776+0000 7f2f7cfa7640 1 -- 172.21.15.57:0/4136826866 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2570823587 0 0) 0x7f2f780af150 con 0x7f2f7814a700 2024-03-29T10:45:53.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.776+0000 7f2f7cfa7640 10 monclient(hunting): _negotiate 2024-03-29T10:45:53.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.776+0000 7f2f7cfa7640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:53.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.776+0000 7f2f7cfa7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:53.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.776+0000 7f2f7cfa7640 10 monclient(hunting): my global_id is 4466 2024-03-29T10:45:53.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.776+0000 7f2f7cfa7640 1 -- 172.21.15.57:0/4136826866 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f64003520 con 0x7f2f7814a700 2024-03-29T10:45:53.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.776+0000 7f2f7cfa7640 1 -- 172.21.15.57:0/4136826866 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920357381 0 0) 0x7f2f780aba70 con 0x7f2f780b0770 2024-03-29T10:45:53.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.776+0000 7f2f7cfa7640 10 monclient(hunting): _negotiate 2024-03-29T10:45:53.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.776+0000 7f2f7cfa7640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:53.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.776+0000 7f2f7cfa7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:53.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.776+0000 7f2f7cfa7640 10 monclient(hunting): my global_id is 4431 2024-03-29T10:45:53.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.776+0000 7f2f7cfa7640 1 -- 172.21.15.57:0/4136826866 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f780af150 con 0x7f2f780b0770 2024-03-29T10:45:53.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.776+0000 7f2f7cfa7640 1 -- 172.21.15.57:0/4136826866 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2765521708 0 0) 0x7f2f64003520 con 0x7f2f7814a700 2024-03-29T10:45:53.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.776+0000 7f2f7cfa7640 1 -- 172.21.15.57:0/4136826866 >> v1:172.21.15.179:6789/0 conn(0x7f2f780ac5a0 legacy=0x7f2f78151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:53.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.778+0000 7f2f7cfa7640 1 -- 172.21.15.57:0/4136826866 >> v1:172.21.15.57:6789/0 conn(0x7f2f780b0770 legacy=0x7f2f78149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:53.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.778+0000 7f2f7cfa7640 10 monclient: _finish_hunting 0 2024-03-29T10:45:53.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.778+0000 7f2f7cfa7640 1 monclient: found mon.noname-b 2024-03-29T10:45:53.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.778+0000 7f2f7cfa7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:53.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.778+0000 7f2f7cfa7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:45:53.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.778+0000 7f2f7cfa7640 1 -- 172.21.15.57:0/4136826866 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f780ac9d0 con 0x7f2f7814a700 2024-03-29T10:45:53.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.778+0000 7f2f7cfa7640 10 monclient: _finish_auth 0 2024-03-29T10:45:53.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.778+0000 7f2f7cfa7640 10 monclient: _check_auth_tickets 2024-03-29T10:45:53.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.778+0000 7f2f7cfa7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:53.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.778+0000 7f2f7cfa7640 1 -- 172.21.15.57:0/4136826866 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f2f68001690 con 0x7f2f7814a700 2024-03-29T10:45:53.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.778+0000 7f2f7cfa7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:53.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.778+0000 7f2f7cfa7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:53.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.778+0000 7f2f7cfa7640 10 monclient: dump: 2024-03-29T10:45:53.763 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:53.764 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:53.764 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:53.764 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:53.764 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:53.764 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:53.764 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:53.764 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:53.764 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:53.764 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:53.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.778+0000 7f2f8020d640 5 monclient: authenticate success, global_id 4466 2024-03-29T10:45:53.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.778+0000 7f2f8020d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:53.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.778+0000 7f2f7cfa7640 1 -- 172.21.15.57:0/4136826866 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f680017f0 con 0x7f2f7814a700 2024-03-29T10:45:53.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.778+0000 7f2f7cfa7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:53.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.778+0000 7f2f7cfa7640 1 -- 172.21.15.57:0/4136826866 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f2f68001c40 con 0x7f2f7814a700 2024-03-29T10:45:53.765 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.778+0000 7f2f8020d640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:53.765 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.778+0000 7f2f8020d640 4 set_mon_vals no callback set 2024-03-29T10:45:53.765 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.778+0000 7f2f7cfa7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:53.765 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.778+0000 7f2f7cfa7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:53.765 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.778+0000 7f2f7cfa7640 10 monclient: dump: 2024-03-29T10:45:53.765 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:53.765 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:53.765 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:53.765 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:53.765 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:53.765 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:53.765 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:53.765 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:53.765 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:53.765 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:53.766 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.779+0000 7f2f8020d640 10 monclient: shutdown 2024-03-29T10:45:53.766 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.779+0000 7f2f8020d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:53.766 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.779+0000 7f2f8020d640 1 -- 172.21.15.57:0/4136826866 >> v1:172.21.15.57:6790/0 conn(0x7f2f7814a700 legacy=0x7f2f780abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:53.766 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.779+0000 7f2f8020d640 1 -- 172.21.15.57:0/4136826866 shutdown_connections 2024-03-29T10:45:53.766 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.779+0000 7f2f8020d640 1 -- 172.21.15.57:0/4136826866 >> 172.21.15.57:0/4136826866 conn(0x7f2f7806b660 msgr2=0x7f2f7806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:53.766 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.779+0000 7f2f8020d640 1 -- 172.21.15.57:0/4136826866 shutdown_connections 2024-03-29T10:45:53.766 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.779+0000 7f2f8020d640 1 -- 172.21.15.57:0/4136826866 wait complete. 2024-03-29T10:45:53.766 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.780+0000 7f2f8020d640 10 monclient: build_initial_monmap 2024-03-29T10:45:53.766 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.780+0000 7f2f8020d640 10 monclient: monmap: 2024-03-29T10:45:53.766 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:45:53.766 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:53.766 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:45:53.766 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:45:53.766 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:53.767 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:53.767 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:45:53.767 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:45:53.767 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:45:53.767 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:53.770 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.781+0000 7f2f8020d640 1 librados: starting msgr at 2024-03-29T10:45:53.770 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.781+0000 7f2f8020d640 1 librados: starting objecter 2024-03-29T10:45:53.770 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.781+0000 7f2f8020d640 1 Processor -- start 2024-03-29T10:45:53.770 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.781+0000 7f2f8020d640 1 -- start start 2024-03-29T10:45:53.771 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.781+0000 7f2f8020d640 1 librados: setting wanted keys 2024-03-29T10:45:53.771 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.781+0000 7f2f8020d640 1 librados: calling monclient init 2024-03-29T10:45:53.771 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.781+0000 7f2f8020d640 10 monclient: init 2024-03-29T10:45:53.771 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.782+0000 7f2f8020d640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:53.771 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.782+0000 7f2f8020d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:45:53.771 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.782+0000 7f2f8020d640 10 monclient(hunting): picked mon.noname-a con 0x7f2f780ac5a0 addr v1:172.21.15.57:6789/0 2024-03-29T10:45:53.771 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.782+0000 7f2f8020d640 10 monclient(hunting): picked mon.noname-c con 0x7f2f780b0770 addr v1:172.21.15.57:6790/0 2024-03-29T10:45:53.771 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.782+0000 7f2f8020d640 10 monclient(hunting): picked mon.noname-b con 0x7f2f7814a700 addr v1:172.21.15.179:6789/0 2024-03-29T10:45:53.771 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.782+0000 7f2f8020d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f78128030 con 0x7f2f780ac5a0 2024-03-29T10:45:53.771 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.782+0000 7f2f8020d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f78125c20 con 0x7f2f7814a700 2024-03-29T10:45:53.771 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.782+0000 7f2f8020d640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f78127110 con 0x7f2f780b0770 2024-03-29T10:45:53.771 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.782+0000 7f2f8020d640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:53.771 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.782+0000 7f2f8020d640 10 monclient(hunting): authenticate will time out at 1305.958381s 2024-03-29T10:45:53.771 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.782+0000 7f2f7dfa9640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f2f780ac5a0 0x7f2f78126690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44226/0 (socket says 172.21.15.57:44226) 2024-03-29T10:45:53.771 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.782+0000 7f2f7dfa9640 1 -- 172.21.15.57:0/3564312379 learned_addr learned my addr 172.21.15.57:0/3564312379 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:45:53.772 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.782+0000 7f2f6e7fc640 10 client.?.objecter ms_handle_connect 0x7f2f780ac5a0 2024-03-29T10:45:53.772 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.782+0000 7f2f6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:53.772 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.783+0000 7f2f6e7fc640 1 -- 172.21.15.57:0/3564312379 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026746993 0 0) 0x7f2f78128030 con 0x7f2f780ac5a0 2024-03-29T10:45:53.772 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.783+0000 7f2f6e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:53.772 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.783+0000 7f2f6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:53.772 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.783+0000 7f2f6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:53.772 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.783+0000 7f2f6e7fc640 10 monclient(hunting): my global_id is 4434 2024-03-29T10:45:53.772 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.783+0000 7f2f6e7fc640 1 -- 172.21.15.57:0/3564312379 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f5c004eb0 con 0x7f2f780ac5a0 2024-03-29T10:45:53.772 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.783+0000 7f2f6e7fc640 10 client.?.objecter ms_handle_connect 0x7f2f780b0770 2024-03-29T10:45:53.772 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.783+0000 7f2f6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:53.772 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.783+0000 7f2f6e7fc640 1 -- 172.21.15.57:0/3564312379 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4027052367 0 0) 0x7f2f5c004eb0 con 0x7f2f780ac5a0 2024-03-29T10:45:53.772 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.783+0000 7f2f6e7fc640 1 -- 172.21.15.57:0/3564312379 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f78128030 con 0x7f2f780ac5a0 2024-03-29T10:45:53.772 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.783+0000 7f2f6e7fc640 1 -- 172.21.15.57:0/3564312379 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f2f70001e90 con 0x7f2f780ac5a0 2024-03-29T10:45:53.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.783+0000 7f2f6e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:53.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.783+0000 7f2f6e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:53.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.783+0000 7f2f6e7fc640 10 monclient(hunting): dump: 2024-03-29T10:45:53.773 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:53.773 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:53.773 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:53.773 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:53.773 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:53.773 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:53.773 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:53.773 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:53.773 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:53.773 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:53.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.783+0000 7f2f6e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:53.773 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.784+0000 7f2f6e7fc640 1 -- 172.21.15.57:0/3564312379 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1710073355 0 0) 0x7f2f78128030 con 0x7f2f780ac5a0 2024-03-29T10:45:53.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.784+0000 7f2f6e7fc640 1 -- 172.21.15.57:0/3564312379 >> v1:172.21.15.57:6790/0 conn(0x7f2f780b0770 legacy=0x7f2f78144db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:53.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.784+0000 7f2f6e7fc640 1 -- 172.21.15.57:0/3564312379 >> v1:172.21.15.179:6789/0 conn(0x7f2f7814a700 legacy=0x7f2f781db600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:53.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.785+0000 7f2f6e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:45:53.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.785+0000 7f2f6e7fc640 1 monclient: found mon.a 2024-03-29T10:45:53.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.785+0000 7f2f6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:53.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.785+0000 7f2f6e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:45:53.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.785+0000 7f2f6e7fc640 1 -- 172.21.15.57:0/3564312379 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f78126da0 con 0x7f2f780ac5a0 2024-03-29T10:45:53.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.785+0000 7f2f6e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:45:53.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.785+0000 7f2f6e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:45:53.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.785+0000 7f2f6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:53.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.785+0000 7f2f8020d640 5 monclient: authenticate success, global_id 4434 2024-03-29T10:45:53.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.785+0000 7f2f8020d640 10 monclient: _renew_subs 2024-03-29T10:45:53.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.785+0000 7f2f8020d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:45:53.774 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.785+0000 7f2f8020d640 1 -- 172.21.15.57:0/3564312379 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f78145a30 con 0x7f2f780ac5a0 2024-03-29T10:45:53.775 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.785+0000 7f2f8020d640 10 client.4434.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:53.775 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.785+0000 7f2f8020d640 10 monclient: _renew_subs 2024-03-29T10:45:53.775 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.785+0000 7f2f8020d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:45:53.775 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.785+0000 7f2f8020d640 1 -- 172.21.15.57:0/3564312379 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f781456a0 con 0x7f2f780ac5a0 2024-03-29T10:45:53.775 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.785+0000 7f2f8020d640 1 librados: init done 2024-03-29T10:45:53.775 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.786+0000 7f2f6e7fc640 1 -- 172.21.15.57:0/3564312379 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f70001ff0 con 0x7f2f780ac5a0 2024-03-29T10:45:53.775 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.786+0000 7f2f6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:53.775 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.786+0000 7f2f6e7fc640 1 -- 172.21.15.57:0/3564312379 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f2f70003e80 con 0x7f2f780ac5a0 2024-03-29T10:45:53.775 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.786+0000 7f2f6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:53.775 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.786+0000 7f2f6e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:53.775 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.786+0000 7f2f6e7fc640 10 monclient: dump: 2024-03-29T10:45:53.775 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:45:53.775 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:45:53.775 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:53.776 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:45:53.776 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:53.776 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:45:53.776 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:45:53.776 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:45:53.776 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:45:53.776 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:45:53.776 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.787+0000 7f2f6f7fe640 4 set_mon_vals no callback set 2024-03-29T10:45:53.776 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.787+0000 7f2f8020d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:53.776 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.787+0000 7f2f8020d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:53.776 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.787+0000 7f2f8020d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:45:53.776 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.787+0000 7f2f8020d640 1 -- 172.21.15.57:0/3564312379 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f60005080 con 0x7f2f780ac5a0 2024-03-29T10:45:53.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.788+0000 7f2f6e7fc640 1 -- 172.21.15.57:0/3564312379 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f2f70003fe0 con 0x7f2f780ac5a0 2024-03-29T10:45:53.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.789+0000 7f2f6e7fc640 1 -- 172.21.15.57:0/3564312379 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f2f70084f90 con 0x7f2f780ac5a0 2024-03-29T10:45:53.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.789+0000 7f2f6e7fc640 10 client.4434.objecter ms_dispatch 0x7f2f78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:45:53.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.789+0000 7f2f6e7fc640 3 client.4434.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:45:53.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.789+0000 7f2f6e7fc640 3 client.4434.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:45:53.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.789+0000 7f2f7d7a8640 10 monclient: get_auth_request con 0x7f2f5c06dd40 auth_method 0 2024-03-29T10:45:53.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.790+0000 7f2f6e7fc640 20 client.4434.objecter dump_active .. 0 homeless 2024-03-29T10:45:53.777 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.790+0000 7f2f6e7fc640 10 client.4434.objecter ms_handle_connect 0x7f2f5c06dd40 2024-03-29T10:45:53.779 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.796+0000 7f2f6e7fc640 1 -- 172.21.15.57:0/3564312379 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2f70050870 con 0x7f2f780ac5a0 2024-03-29T10:45:53.779 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.796+0000 7f2f6e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:53.779 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:53.796+0000 7f2f6e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:54.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:54.128+0000 7f2f8020d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:54.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:54.128+0000 7f2f8020d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:54.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:54.128+0000 7f2f8020d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:45:54.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:54.128+0000 7f2f8020d640 1 -- 172.21.15.57:0/3564312379 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f60005330 con 0x7f2f780ac5a0 2024-03-29T10:45:54.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:54.128+0000 7f2f6e7fc640 1 -- 172.21.15.57:0/3564312379 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f2f700527a0 con 0x7f2f780ac5a0 2024-03-29T10:45:54.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:54.128+0000 7f2f6e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:54.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:54.128+0000 7f2f6e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:54.114 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:45:54.114 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:54.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:54.134+0000 7f2f8020d640 10 librados: watch_flush enter 2024-03-29T10:45:54.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:54.134+0000 7f2f8020d640 10 librados: watch_flush exit 2024-03-29T10:45:54.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:54.134+0000 7f2f8020d640 20 client.4434.objecter shutdown clearing up homeless session... 2024-03-29T10:45:54.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:54.134+0000 7f2f8020d640 10 client.4434.objecter successfully canceled tick 2024-03-29T10:45:54.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:54.134+0000 7f2f8020d640 1 -- 172.21.15.57:0/3564312379 >> v1:172.21.15.179:6816/2698295936 conn(0x7f2f5c06dd40 legacy=0x7f2f5c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:54.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:54.134+0000 7f2f8020d640 10 monclient: shutdown 2024-03-29T10:45:54.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:54.134+0000 7f2f8020d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:54.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:54.134+0000 7f2f8020d640 1 -- 172.21.15.57:0/3564312379 >> v1:172.21.15.57:6789/0 conn(0x7f2f780ac5a0 legacy=0x7f2f78126690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:54.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:54.134+0000 7f2f8020d640 1 -- 172.21.15.57:0/3564312379 shutdown_connections 2024-03-29T10:45:54.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:54.134+0000 7f2f8020d640 1 -- 172.21.15.57:0/3564312379 >> 172.21.15.57:0/3564312379 conn(0x7f2f7806b660 msgr2=0x7f2f78071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:54.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:54.135+0000 7f2f8020d640 1 -- 172.21.15.57:0/3564312379 shutdown_connections 2024-03-29T10:45:54.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:54.135+0000 7f2f8020d640 1 -- 172.21.15.57:0/3564312379 wait complete. 2024-03-29T10:45:54.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:45:54.135+0000 7f2f8020d640 1 librados: shutdown 2024-03-29T10:45:54.134 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:00.941 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:01.096 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.113+0000 7f7616c93640 10 monclient: get_monmap_and_config 2024-03-29T10:46:01.096 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.113+0000 7f7616c93640 10 monclient: build_initial_monmap 2024-03-29T10:46:01.096 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.113+0000 7f7616c93640 10 monclient: monmap: 2024-03-29T10:46:01.096 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:46:01.096 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:01.096 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:46:01.114168+0000 2024-03-29T10:46:01.096 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:46:01.114168+0000 2024-03-29T10:46:01.096 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:01.097 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:01.097 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:46:01.097 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:46:01.097 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:46:01.097 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:01.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.114+0000 7f7616c93640 1 Processor -- start 2024-03-29T10:46:01.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.114+0000 7f7616c93640 1 -- start start 2024-03-29T10:46:01.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.114+0000 7f7616c93640 10 monclient: init 2024-03-29T10:46:01.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.115+0000 7f7616c93640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:01.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.115+0000 7f7616c93640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:46:01.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.115+0000 7f7616c93640 10 monclient(hunting): picked mon.noname-c con 0x7f76101422b0 addr v1:172.21.15.179:6789/0 2024-03-29T10:46:01.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.115+0000 7f7616c93640 10 monclient(hunting): picked mon.noname-a con 0x7f7610142690 addr v1:172.21.15.57:6789/0 2024-03-29T10:46:01.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.115+0000 7f760ffff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f7610142690 0x7f76100b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:56758/0 (socket says 172.21.15.57:56758) 2024-03-29T10:46:01.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.115+0000 7f760ffff640 1 -- 172.21.15.57:0/3817958778 learned_addr learned my addr 172.21.15.57:0/3817958778 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:46:01.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.115+0000 7f7616c93640 10 monclient(hunting): picked mon.noname-b con 0x7f76100b17d0 addr v1:172.21.15.57:6790/0 2024-03-29T10:46:01.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.115+0000 7f7616c93640 1 -- 172.21.15.57:0/3817958778 --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7610142c80 con 0x7f7610142690 2024-03-29T10:46:01.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.115+0000 7f7616c93640 1 -- 172.21.15.57:0/3817958778 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76100aae40 con 0x7f76101422b0 2024-03-29T10:46:01.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.115+0000 7f7616c93640 1 -- 172.21.15.57:0/3817958778 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761013fbd0 con 0x7f76100b17d0 2024-03-29T10:46:01.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.116+0000 7f7616c93640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:01.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.116+0000 7f7616c93640 10 monclient(hunting): authenticate will time out at 1313.292070s 2024-03-29T10:46:01.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.116+0000 7f760f7fe640 1 -- 172.21.15.57:0/3817958778 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699548318 0 0) 0x7f76100aae40 con 0x7f76101422b0 2024-03-29T10:46:01.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.116+0000 7f760f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:46:01.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.116+0000 7f760f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:01.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.116+0000 7f760f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:01.105 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.116+0000 7f760f7fe640 10 monclient(hunting): my global_id is 4435 2024-03-29T10:46:01.106 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.116+0000 7f760f7fe640 1 -- 172.21.15.57:0/3817958778 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75f8003520 con 0x7f76101422b0 2024-03-29T10:46:01.106 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.116+0000 7f760f7fe640 1 -- 172.21.15.57:0/3817958778 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906002496 0 0) 0x7f761013fbd0 con 0x7f76100b17d0 2024-03-29T10:46:01.106 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.116+0000 7f760f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:46:01.106 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.117+0000 7f760f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:01.106 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.117+0000 7f760f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:01.106 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.117+0000 7f760f7fe640 10 monclient(hunting): my global_id is 4469 2024-03-29T10:46:01.106 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.117+0000 7f760f7fe640 1 -- 172.21.15.57:0/3817958778 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76100aae40 con 0x7f76100b17d0 2024-03-29T10:46:01.106 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.117+0000 7f760f7fe640 1 -- 172.21.15.57:0/3817958778 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608992648 0 0) 0x7f7610142c80 con 0x7f7610142690 2024-03-29T10:46:01.106 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.117+0000 7f760f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:46:01.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.117+0000 7f760f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:01.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.117+0000 7f760f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:01.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.117+0000 7f760f7fe640 10 monclient(hunting): my global_id is 4437 2024-03-29T10:46:01.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.117+0000 7f760f7fe640 1 -- 172.21.15.57:0/3817958778 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f761013fbd0 con 0x7f7610142690 2024-03-29T10:46:01.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.117+0000 7f760f7fe640 1 -- 172.21.15.57:0/3817958778 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 714126468 0 0) 0x7f75f8003520 con 0x7f76101422b0 2024-03-29T10:46:01.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.117+0000 7f760f7fe640 1 -- 172.21.15.57:0/3817958778 >> v1:172.21.15.57:6790/0 conn(0x7f76100b17d0 legacy=0x7f76100b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:01.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.117+0000 7f760f7fe640 1 -- 172.21.15.57:0/3817958778 >> v1:172.21.15.57:6789/0 conn(0x7f7610142690 legacy=0x7f76100b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:01.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.117+0000 7f760f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:46:01.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.117+0000 7f760f7fe640 1 monclient: found mon.noname-c 2024-03-29T10:46:01.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.117+0000 7f760f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:01.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.117+0000 7f760f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:46:01.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.117+0000 7f760f7fe640 1 -- 172.21.15.57:0/3817958778 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7610151940 con 0x7f76101422b0 2024-03-29T10:46:01.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.117+0000 7f760f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:46:01.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.117+0000 7f760f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:46:01.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.118+0000 7f760f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:01.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.118+0000 7f760f7fe640 1 -- 172.21.15.57:0/3817958778 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f7600001690 con 0x7f76101422b0 2024-03-29T10:46:01.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.118+0000 7f7616c93640 5 monclient: authenticate success, global_id 4435 2024-03-29T10:46:01.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.118+0000 7f7616c93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:01.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.118+0000 7f760f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:01.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.118+0000 7f760f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:01.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.118+0000 7f760f7fe640 10 monclient: dump: 2024-03-29T10:46:01.108 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:01.108 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:01.108 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:01.108 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:01.108 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:01.108 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:01.108 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:01.108 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:01.109 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:01.109 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:01.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.118+0000 7f760f7fe640 1 -- 172.21.15.57:0/3817958778 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76000018b0 con 0x7f76101422b0 2024-03-29T10:46:01.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.118+0000 7f7616c93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:01.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.118+0000 7f760f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:01.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.118+0000 7f760f7fe640 1 -- 172.21.15.57:0/3817958778 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f7600001bc0 con 0x7f76101422b0 2024-03-29T10:46:01.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.118+0000 7f7616c93640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:01.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.118+0000 7f7616c93640 4 set_mon_vals no callback set 2024-03-29T10:46:01.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.118+0000 7f760f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:01.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.118+0000 7f760f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:01.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.118+0000 7f760f7fe640 10 monclient: dump: 2024-03-29T10:46:01.109 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:01.109 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:01.109 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:01.110 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:01.110 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:01.110 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:01.110 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:01.110 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:01.110 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:01.110 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:01.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.119+0000 7f7616c93640 10 monclient: shutdown 2024-03-29T10:46:01.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.119+0000 7f7616c93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:01.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.119+0000 7f7616c93640 1 -- 172.21.15.57:0/3817958778 >> v1:172.21.15.179:6789/0 conn(0x7f76101422b0 legacy=0x7f761013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:01.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.119+0000 7f7616c93640 1 -- 172.21.15.57:0/3817958778 shutdown_connections 2024-03-29T10:46:01.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.119+0000 7f7616c93640 1 -- 172.21.15.57:0/3817958778 >> 172.21.15.57:0/3817958778 conn(0x7f7610066a00 msgr2=0x7f7610066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:01.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.119+0000 7f7616c93640 1 -- 172.21.15.57:0/3817958778 shutdown_connections 2024-03-29T10:46:01.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.119+0000 7f7616c93640 1 -- 172.21.15.57:0/3817958778 wait complete. 2024-03-29T10:46:01.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.121+0000 7f7616c93640 10 monclient: build_initial_monmap 2024-03-29T10:46:01.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.121+0000 7f7616c93640 10 monclient: monmap: 2024-03-29T10:46:01.111 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:46:01.111 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:01.111 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:46:01.111 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:46:01.111 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:01.111 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:01.111 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:46:01.111 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:46:01.111 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:46:01.111 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:01.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.122+0000 7f7616c93640 1 librados: starting msgr at 2024-03-29T10:46:01.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.122+0000 7f7616c93640 1 librados: starting objecter 2024-03-29T10:46:01.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.122+0000 7f7616c93640 1 Processor -- start 2024-03-29T10:46:01.116 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.122+0000 7f7616c93640 1 -- start start 2024-03-29T10:46:01.116 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.122+0000 7f7616c93640 1 librados: setting wanted keys 2024-03-29T10:46:01.116 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.122+0000 7f7616c93640 1 librados: calling monclient init 2024-03-29T10:46:01.116 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.122+0000 7f7616c93640 10 monclient: init 2024-03-29T10:46:01.116 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.124+0000 7f7616c93640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:01.116 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.124+0000 7f7616c93640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:01.116 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.124+0000 7f7616c93640 10 monclient(hunting): picked mon.noname-a con 0x7f76100b17d0 addr v1:172.21.15.57:6789/0 2024-03-29T10:46:01.116 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.124+0000 7f7614a2f640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f76100b17d0 0x7f76100af0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:57262/0 (socket says 172.21.15.57:57262) 2024-03-29T10:46:01.116 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.124+0000 7f7614a2f640 1 -- 172.21.15.57:0/2824788462 learned_addr learned my addr 172.21.15.57:0/2824788462 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:46:01.116 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.125+0000 7f7616c93640 10 monclient(hunting): picked mon.noname-b con 0x7f76101422b0 addr v1:172.21.15.179:6789/0 2024-03-29T10:46:01.116 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.125+0000 7f7616c93640 10 monclient(hunting): picked mon.noname-c con 0x7f7610142690 addr v1:172.21.15.57:6790/0 2024-03-29T10:46:01.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.125+0000 7f7616c93640 1 -- 172.21.15.57:0/2824788462 --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76100b0180 con 0x7f76100b17d0 2024-03-29T10:46:01.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.125+0000 7f7616c93640 1 -- 172.21.15.57:0/2824788462 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76100b0c20 con 0x7f76101422b0 2024-03-29T10:46:01.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.125+0000 7f7616c93640 1 -- 172.21.15.57:0/2824788462 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76100b0f00 con 0x7f7610142690 2024-03-29T10:46:01.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.125+0000 7f7616c93640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:01.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.125+0000 7f7616c93640 10 monclient(hunting): authenticate will time out at 1313.301346s 2024-03-29T10:46:01.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.125+0000 7f760d7fa640 10 client.?.objecter ms_handle_connect 0x7f76100b17d0 2024-03-29T10:46:01.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.125+0000 7f760d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:01.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.125+0000 7f760d7fa640 10 client.?.objecter ms_handle_connect 0x7f76101422b0 2024-03-29T10:46:01.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.125+0000 7f760d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:01.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.125+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3472154383 0 0) 0x7f76100b0180 con 0x7f76100b17d0 2024-03-29T10:46:01.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:46:01.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:01.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:01.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 10 monclient(hunting): my global_id is 4440 2024-03-29T10:46:01.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75fc004eb0 con 0x7f76100b17d0 2024-03-29T10:46:01.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1465833124 0 0) 0x7f76100b0c20 con 0x7f76101422b0 2024-03-29T10:46:01.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:46:01.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:01.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:01.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 10 monclient(hunting): my global_id is 4438 2024-03-29T10:46:01.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76100b0180 con 0x7f76101422b0 2024-03-29T10:46:01.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2761359031 0 0) 0x7f75fc004eb0 con 0x7f76100b17d0 2024-03-29T10:46:01.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76100b0c20 con 0x7f76100b17d0 2024-03-29T10:46:01.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 10 client.?.objecter ms_handle_connect 0x7f7610142690 2024-03-29T10:46:01.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:01.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f7600001690 con 0x7f76100b17d0 2024-03-29T10:46:01.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:01.118 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:01.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 10 monclient(hunting): dump: 2024-03-29T10:46:01.119 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:01.119 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:01.119 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:01.119 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:01.119 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:01.119 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:01.119 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:01.119 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:01.119 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:01.119 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:01.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:01.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1238410422 0 0) 0x7f76100b0180 con 0x7f76101422b0 2024-03-29T10:46:01.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75fc004eb0 con 0x7f76101422b0 2024-03-29T10:46:01.119 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f76040016f0 con 0x7f76101422b0 2024-03-29T10:46:01.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:01.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:01.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.126+0000 7f760d7fa640 10 monclient(hunting): dump: 2024-03-29T10:46:01.120 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:01.120 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:01.120 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:01.120 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:01.120 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:01.120 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:01.120 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:01.120 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:01.120 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:01.120 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:01.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.127+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508790650 0 0) 0x7f76100b0f00 con 0x7f7610142690 2024-03-29T10:46:01.120 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.127+0000 7f760d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:46:01.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.127+0000 7f760d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:01.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.127+0000 7f760d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:01.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.127+0000 7f760d7fa640 10 monclient(hunting): my global_id is 4472 2024-03-29T10:46:01.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.127+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76100b0180 con 0x7f7610142690 2024-03-29T10:46:01.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.127+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1330777120 0 0) 0x7f76100b0180 con 0x7f7610142690 2024-03-29T10:46:01.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.127+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76100b0f00 con 0x7f7610142690 2024-03-29T10:46:01.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.127+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f7608001cb0 con 0x7f7610142690 2024-03-29T10:46:01.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.127+0000 7f760d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:01.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.127+0000 7f760d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:01.121 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.127+0000 7f760d7fa640 10 monclient(hunting): dump: 2024-03-29T10:46:01.121 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:01.121 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:01.121 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:01.121 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:01.122 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:01.122 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:01.122 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:01.122 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:01.122 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:01.122 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:01.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.127+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3662057011 0 0) 0x7f76100b0c20 con 0x7f76100b17d0 2024-03-29T10:46:01.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.127+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 >> v1:172.21.15.57:6790/0 conn(0x7f7610142690 legacy=0x7f76101db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:01.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.127+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 >> v1:172.21.15.179:6789/0 conn(0x7f76101422b0 legacy=0x7f76101daa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:01.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.127+0000 7f760d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:46:01.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.127+0000 7f760d7fa640 1 monclient: found mon.a 2024-03-29T10:46:01.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.127+0000 7f760d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:01.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.127+0000 7f760d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:46:01.122 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.127+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76100b11e0 con 0x7f76100b17d0 2024-03-29T10:46:01.123 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.127+0000 7f760d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:46:01.123 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.127+0000 7f760d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:46:01.123 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.127+0000 7f760d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:01.123 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.128+0000 7f7616c93640 5 monclient: authenticate success, global_id 4440 2024-03-29T10:46:01.123 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.128+0000 7f7616c93640 10 monclient: _renew_subs 2024-03-29T10:46:01.123 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.128+0000 7f7616c93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:46:01.123 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.128+0000 7f7616c93640 1 -- 172.21.15.57:0/2824788462 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76100afd70 con 0x7f76100b17d0 2024-03-29T10:46:01.123 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.128+0000 7f7616c93640 10 client.4440.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:01.123 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.128+0000 7f7616c93640 10 monclient: _renew_subs 2024-03-29T10:46:01.123 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.128+0000 7f7616c93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:46:01.123 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.128+0000 7f7616c93640 1 -- 172.21.15.57:0/2824788462 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76101e3030 con 0x7f76100b17d0 2024-03-29T10:46:01.123 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.128+0000 7f7616c93640 1 librados: init done 2024-03-29T10:46:01.123 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.130+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7600001270 con 0x7f76100b17d0 2024-03-29T10:46:01.123 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.130+0000 7f760d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:01.124 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.130+0000 7f760effd640 4 set_mon_vals no callback set 2024-03-29T10:46:01.124 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.130+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f7600003a40 con 0x7f76100b17d0 2024-03-29T10:46:01.124 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.130+0000 7f760d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:01.124 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.130+0000 7f760d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:01.124 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.130+0000 7f760d7fa640 10 monclient: dump: 2024-03-29T10:46:01.124 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:01.124 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:01.124 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:01.124 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:01.124 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:01.124 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:01.124 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:01.124 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:01.124 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:01.124 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:01.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.131+0000 7f7616c93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:01.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.131+0000 7f7616c93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:01.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.131+0000 7f7616c93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:46:01.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.131+0000 7f7616c93640 1 -- 172.21.15.57:0/2824788462 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f0005080 con 0x7f76100b17d0 2024-03-29T10:46:01.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.132+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f7600003ba0 con 0x7f76100b17d0 2024-03-29T10:46:01.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.133+0000 7f760ffff640 10 monclient: get_auth_request con 0x7f75fc06e170 auth_method 0 2024-03-29T10:46:01.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.133+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f7600084bb0 con 0x7f76100b17d0 2024-03-29T10:46:01.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.133+0000 7f760d7fa640 10 client.4440.objecter ms_dispatch 0x7f7610000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:01.125 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.133+0000 7f760d7fa640 3 client.4440.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:01.126 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.133+0000 7f760d7fa640 3 client.4440.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:01.126 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.133+0000 7f760d7fa640 20 client.4440.objecter dump_active .. 0 homeless 2024-03-29T10:46:01.126 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.133+0000 7f760d7fa640 10 client.4440.objecter ms_handle_connect 0x7f75fc06e170 2024-03-29T10:46:01.128 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.145+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7600050490 con 0x7f76100b17d0 2024-03-29T10:46:01.128 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.145+0000 7f760d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:01.128 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.145+0000 7f760d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:01.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.600+0000 7f7616c93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:01.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.600+0000 7f7616c93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:01.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.600+0000 7f7616c93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:46:01.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.600+0000 7f7616c93640 1 -- 172.21.15.57:0/2824788462 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75f0005330 con 0x7f76100b17d0 2024-03-29T10:46:01.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.601+0000 7f760d7fa640 1 -- 172.21.15.57:0/2824788462 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f76000523c0 con 0x7f76100b17d0 2024-03-29T10:46:01.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.601+0000 7f760d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:01.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.601+0000 7f760d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:01.584 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:46:01.584 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:01.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.605+0000 7f7616c93640 10 librados: watch_flush enter 2024-03-29T10:46:01.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.605+0000 7f7616c93640 10 librados: watch_flush exit 2024-03-29T10:46:01.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.605+0000 7f7616c93640 20 client.4440.objecter shutdown clearing up homeless session... 2024-03-29T10:46:01.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.605+0000 7f7616c93640 10 client.4440.objecter successfully canceled tick 2024-03-29T10:46:01.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.605+0000 7f7616c93640 1 -- 172.21.15.57:0/2824788462 >> v1:172.21.15.179:6816/2698295936 conn(0x7f75fc06e170 legacy=0x7f75fc070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:01.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.605+0000 7f7616c93640 10 monclient: shutdown 2024-03-29T10:46:01.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.605+0000 7f7616c93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:01.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.605+0000 7f7616c93640 1 -- 172.21.15.57:0/2824788462 >> v1:172.21.15.57:6789/0 conn(0x7f76100b17d0 legacy=0x7f76100af0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:01.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.605+0000 7f7616c93640 1 -- 172.21.15.57:0/2824788462 shutdown_connections 2024-03-29T10:46:01.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.605+0000 7f7616c93640 1 -- 172.21.15.57:0/2824788462 >> 172.21.15.57:0/2824788462 conn(0x7f7610066a00 msgr2=0x7f76100b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:01.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.606+0000 7f7616c93640 1 -- 172.21.15.57:0/2824788462 shutdown_connections 2024-03-29T10:46:01.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.606+0000 7f7616c93640 1 -- 172.21.15.57:0/2824788462 wait complete. 2024-03-29T10:46:01.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:01.606+0000 7f7616c93640 1 librados: shutdown 2024-03-29T10:46:01.602 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:08.409 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:08.578 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.595+0000 7f681d93b640 10 monclient: get_monmap_and_config 2024-03-29T10:46:08.579 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.595+0000 7f681d93b640 10 monclient: build_initial_monmap 2024-03-29T10:46:08.579 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.595+0000 7f681d93b640 10 monclient: monmap: 2024-03-29T10:46:08.579 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:46:08.579 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:08.579 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:46:08.595984+0000 2024-03-29T10:46:08.579 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:46:08.595984+0000 2024-03-29T10:46:08.579 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:08.579 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:08.579 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:46:08.579 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:46:08.579 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:46:08.579 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:08.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.596+0000 7f681d93b640 1 Processor -- start 2024-03-29T10:46:08.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.596+0000 7f681d93b640 1 -- start start 2024-03-29T10:46:08.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.596+0000 7f681d93b640 10 monclient: init 2024-03-29T10:46:08.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.597+0000 7f681d93b640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:08.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.597+0000 7f681d93b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:08.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.597+0000 7f681d93b640 10 monclient(hunting): picked mon.noname-a con 0x7f68180a9930 addr v1:172.21.15.57:6789/0 2024-03-29T10:46:08.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.597+0000 7f681d93b640 10 monclient(hunting): picked mon.noname-b con 0x7f6818149dc0 addr v1:172.21.15.57:6790/0 2024-03-29T10:46:08.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.597+0000 7f681d93b640 10 monclient(hunting): picked mon.noname-c con 0x7f681814ff40 addr v1:172.21.15.179:6789/0 2024-03-29T10:46:08.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.597+0000 7f681d93b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68180aae60 con 0x7f68180a9930 2024-03-29T10:46:08.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.597+0000 7f681d93b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68180b4310 con 0x7f681814ff40 2024-03-29T10:46:08.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.597+0000 7f681d93b640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68180afdd0 con 0x7f6818149dc0 2024-03-29T10:46:08.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.597+0000 7f681d93b640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:08.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.597+0000 7f681d93b640 10 monclient(hunting): authenticate will time out at 1320.773677s 2024-03-29T10:46:08.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.602+0000 7f6816ffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f68180a9930 0x7f6818143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44409/0 (socket says 172.21.15.57:44409) 2024-03-29T10:46:08.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.602+0000 7f6816ffd640 1 -- 172.21.15.57:0/1179064712 learned_addr learned my addr 172.21.15.57:0/1179064712 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:46:08.592 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.604+0000 7f6815ffb640 1 -- 172.21.15.57:0/1179064712 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125428944 0 0) 0x7f68180aae60 con 0x7f68180a9930 2024-03-29T10:46:08.592 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.604+0000 7f6815ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:08.592 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.604+0000 7f6815ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:08.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.604+0000 7f6815ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:08.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.604+0000 7f6815ffb640 10 monclient(hunting): my global_id is 4443 2024-03-29T10:46:08.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.604+0000 7f6815ffb640 1 -- 172.21.15.57:0/1179064712 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6800003520 con 0x7f68180a9930 2024-03-29T10:46:08.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.609+0000 7f6815ffb640 1 -- 172.21.15.57:0/1179064712 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2044229096 0 0) 0x7f68180afdd0 con 0x7f6818149dc0 2024-03-29T10:46:08.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.609+0000 7f6815ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:08.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.609+0000 7f6815ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:08.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.609+0000 7f6815ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:08.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.609+0000 7f6815ffb640 10 monclient(hunting): my global_id is 4475 2024-03-29T10:46:08.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.609+0000 7f6815ffb640 1 -- 172.21.15.57:0/1179064712 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6800003840 con 0x7f6818149dc0 2024-03-29T10:46:08.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.609+0000 7f6815ffb640 1 -- 172.21.15.57:0/1179064712 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087536731 0 0) 0x7f68180b4310 con 0x7f681814ff40 2024-03-29T10:46:08.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.609+0000 7f6815ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:08.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.609+0000 7f6815ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:08.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.609+0000 7f6815ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:08.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.609+0000 7f6815ffb640 10 monclient(hunting): my global_id is 4441 2024-03-29T10:46:08.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.609+0000 7f6815ffb640 1 -- 172.21.15.57:0/1179064712 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68180afdd0 con 0x7f681814ff40 2024-03-29T10:46:08.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.610+0000 7f6815ffb640 1 -- 172.21.15.57:0/1179064712 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 945215238 0 0) 0x7f6800003520 con 0x7f68180a9930 2024-03-29T10:46:08.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.610+0000 7f6815ffb640 1 -- 172.21.15.57:0/1179064712 >> v1:172.21.15.57:6790/0 conn(0x7f6818149dc0 legacy=0x7f681814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:08.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.610+0000 7f6815ffb640 1 -- 172.21.15.57:0/1179064712 >> v1:172.21.15.179:6789/0 conn(0x7f681814ff40 legacy=0x7f6818152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:08.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.610+0000 7f6815ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:46:08.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.610+0000 7f6815ffb640 1 monclient: found mon.noname-a 2024-03-29T10:46:08.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.610+0000 7f6815ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:08.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.610+0000 7f6815ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:46:08.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.610+0000 7f6815ffb640 1 -- 172.21.15.57:0/1179064712 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681814a950 con 0x7f68180a9930 2024-03-29T10:46:08.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.610+0000 7f6815ffb640 10 monclient: _finish_auth 0 2024-03-29T10:46:08.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.610+0000 7f6815ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:46:08.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.610+0000 7f6815ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:08.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.610+0000 7f681d93b640 5 monclient: authenticate success, global_id 4443 2024-03-29T10:46:08.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.610+0000 7f6815ffb640 1 -- 172.21.15.57:0/1179064712 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f680c001670 con 0x7f68180a9930 2024-03-29T10:46:08.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.610+0000 7f681d93b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:08.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.610+0000 7f6815ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:08.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.610+0000 7f6815ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:08.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.610+0000 7f6815ffb640 10 monclient: dump: 2024-03-29T10:46:08.595 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:08.595 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:08.595 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:08.595 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:08.595 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:08.595 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:08.595 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:08.595 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:08.595 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:08.596 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:08.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.610+0000 7f681d93b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:08.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.610+0000 7f6815ffb640 1 -- 172.21.15.57:0/1179064712 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f680c001850 con 0x7f68180a9930 2024-03-29T10:46:08.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.610+0000 7f6815ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:08.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.610+0000 7f6815ffb640 1 -- 172.21.15.57:0/1179064712 <== mon.0 v1:172.21.15.57:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f680c001c60 con 0x7f68180a9930 2024-03-29T10:46:08.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.610+0000 7f6815ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:08.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.610+0000 7f6815ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:08.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.610+0000 7f6815ffb640 10 monclient: dump: 2024-03-29T10:46:08.597 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:08.597 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:08.597 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:08.597 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:08.598 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:08.598 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:08.598 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:08.598 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:08.598 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:08.598 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:08.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.611+0000 7f681d93b640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:08.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.611+0000 7f681d93b640 4 set_mon_vals no callback set 2024-03-29T10:46:08.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.612+0000 7f681d93b640 10 monclient: shutdown 2024-03-29T10:46:08.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.612+0000 7f681d93b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:08.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.612+0000 7f681d93b640 1 -- 172.21.15.57:0/1179064712 >> v1:172.21.15.57:6789/0 conn(0x7f68180a9930 legacy=0x7f6818143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:08.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.613+0000 7f681d93b640 1 -- 172.21.15.57:0/1179064712 shutdown_connections 2024-03-29T10:46:08.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.613+0000 7f681d93b640 1 -- 172.21.15.57:0/1179064712 >> 172.21.15.57:0/1179064712 conn(0x7f68180b35e0 msgr2=0x7f68180b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:08.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.616+0000 7f681d93b640 1 -- 172.21.15.57:0/1179064712 shutdown_connections 2024-03-29T10:46:08.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.616+0000 7f681d93b640 1 -- 172.21.15.57:0/1179064712 wait complete. 2024-03-29T10:46:08.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.618+0000 7f681d93b640 10 monclient: build_initial_monmap 2024-03-29T10:46:08.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.618+0000 7f681d93b640 10 monclient: monmap: 2024-03-29T10:46:08.600 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:46:08.600 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:08.600 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:46:08.600 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:46:08.600 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:08.601 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:08.601 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:46:08.601 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:46:08.601 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:46:08.601 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:08.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.618+0000 7f681d93b640 1 librados: starting msgr at 2024-03-29T10:46:08.602 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.618+0000 7f681d93b640 1 librados: starting objecter 2024-03-29T10:46:08.602 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.618+0000 7f681d93b640 1 Processor -- start 2024-03-29T10:46:08.602 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.619+0000 7f681d93b640 1 -- start start 2024-03-29T10:46:08.602 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.619+0000 7f681d93b640 1 librados: setting wanted keys 2024-03-29T10:46:08.602 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.619+0000 7f681d93b640 1 librados: calling monclient init 2024-03-29T10:46:08.602 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.619+0000 7f681d93b640 10 monclient: init 2024-03-29T10:46:08.605 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.623+0000 7f681d93b640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:08.605 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.623+0000 7f681d93b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:08.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.625+0000 7f681d93b640 10 monclient(hunting): picked mon.noname-a con 0x7f68180a9930 addr v1:172.21.15.57:6789/0 2024-03-29T10:46:08.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.625+0000 7f681d93b640 10 monclient(hunting): picked mon.noname-c con 0x7f6818149dc0 addr v1:172.21.15.57:6790/0 2024-03-29T10:46:08.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.625+0000 7f681d93b640 10 monclient(hunting): picked mon.noname-b con 0x7f681814ff40 addr v1:172.21.15.179:6789/0 2024-03-29T10:46:08.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.625+0000 7f681d93b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f681813cc10 con 0x7f68180a9930 2024-03-29T10:46:08.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.625+0000 7f681d93b640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f681813db50 con 0x7f681814ff40 2024-03-29T10:46:08.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.625+0000 7f681d93b640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f681806ebe0 con 0x7f6818149dc0 2024-03-29T10:46:08.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.625+0000 7f681d93b640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:08.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.625+0000 7f681d93b640 10 monclient(hunting): authenticate will time out at 1320.801716s 2024-03-29T10:46:08.615 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.631+0000 7f68177fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f681814ff40 0x7f68181eeaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:58405/0 (socket says 172.21.15.57:58405) 2024-03-29T10:46:08.616 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.631+0000 7f68177fe640 1 -- 172.21.15.57:0/3959849770 learned_addr learned my addr 172.21.15.57:0/3959849770 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:46:08.616 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.631+0000 7f68167fc640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f6818149dc0 0x7f68181e7e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:54191/0 (socket says 172.21.15.57:54191) 2024-03-29T10:46:08.620 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.633+0000 7f67fffff640 10 client.?.objecter ms_handle_connect 0x7f6818149dc0 2024-03-29T10:46:08.620 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.634+0000 7f67fffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:08.620 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.634+0000 7f67fffff640 10 client.?.objecter ms_handle_connect 0x7f681814ff40 2024-03-29T10:46:08.620 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.634+0000 7f67fffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:08.620 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.634+0000 7f67fffff640 10 client.?.objecter ms_handle_connect 0x7f68180a9930 2024-03-29T10:46:08.620 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.634+0000 7f67fffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:08.620 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.634+0000 7f67fffff640 1 -- 172.21.15.57:0/3959849770 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3011847729 0 0) 0x7f681813cc10 con 0x7f68180a9930 2024-03-29T10:46:08.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.634+0000 7f67fffff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:08.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.634+0000 7f67fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:08.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.634+0000 7f67fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:08.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.634+0000 7f67fffff640 10 monclient(hunting): my global_id is 4446 2024-03-29T10:46:08.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.634+0000 7f67fffff640 1 -- 172.21.15.57:0/3959849770 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6804004eb0 con 0x7f68180a9930 2024-03-29T10:46:08.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.634+0000 7f67fffff640 1 -- 172.21.15.57:0/3959849770 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961677943 0 0) 0x7f681813db50 con 0x7f681814ff40 2024-03-29T10:46:08.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.634+0000 7f67fffff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:08.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.634+0000 7f67fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:08.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.634+0000 7f67fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:08.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.634+0000 7f67fffff640 10 monclient(hunting): my global_id is 4444 2024-03-29T10:46:08.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.634+0000 7f67fffff640 1 -- 172.21.15.57:0/3959849770 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f681813cc10 con 0x7f681814ff40 2024-03-29T10:46:08.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.635+0000 7f67fffff640 1 -- 172.21.15.57:0/3959849770 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3973409453 0 0) 0x7f6804004eb0 con 0x7f68180a9930 2024-03-29T10:46:08.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.635+0000 7f67fffff640 1 -- 172.21.15.57:0/3959849770 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f681813db50 con 0x7f68180a9930 2024-03-29T10:46:08.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.635+0000 7f67fffff640 1 -- 172.21.15.57:0/3959849770 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2606095881 0 0) 0x7f681813cc10 con 0x7f681814ff40 2024-03-29T10:46:08.622 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.635+0000 7f67fffff640 1 -- 172.21.15.57:0/3959849770 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6804004eb0 con 0x7f681814ff40 2024-03-29T10:46:08.622 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.635+0000 7f67fffff640 1 -- 172.21.15.57:0/3959849770 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f680c001c60 con 0x7f68180a9930 2024-03-29T10:46:08.622 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.635+0000 7f67fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:08.622 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.635+0000 7f67fffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:08.622 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.635+0000 7f67fffff640 10 monclient(hunting): dump: 2024-03-29T10:46:08.622 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:08.622 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:08.622 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:08.622 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:08.622 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:08.622 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:08.622 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:08.622 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:08.623 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:08.623 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:08.623 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.635+0000 7f67fffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:08.623 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.635+0000 7f67fffff640 1 -- 172.21.15.57:0/3959849770 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f6810001b50 con 0x7f681814ff40 2024-03-29T10:46:08.623 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.635+0000 7f67fffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:08.623 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.635+0000 7f67fffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:08.623 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.635+0000 7f67fffff640 10 monclient(hunting): dump: 2024-03-29T10:46:08.623 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:08.623 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:08.623 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:08.623 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:08.623 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:08.623 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:08.623 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:08.623 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:08.624 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:08.624 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:08.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.635+0000 7f67fffff640 1 -- 172.21.15.57:0/3959849770 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1748039529 0 0) 0x7f681806ebe0 con 0x7f6818149dc0 2024-03-29T10:46:08.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.635+0000 7f67fffff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:08.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.635+0000 7f67fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:08.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.635+0000 7f67fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:08.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.635+0000 7f67fffff640 10 monclient(hunting): my global_id is 4478 2024-03-29T10:46:08.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.635+0000 7f67fffff640 1 -- 172.21.15.57:0/3959849770 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f681813cc10 con 0x7f6818149dc0 2024-03-29T10:46:08.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.636+0000 7f67fffff640 1 -- 172.21.15.57:0/3959849770 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 634433189 0 0) 0x7f6804004eb0 con 0x7f681814ff40 2024-03-29T10:46:08.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.636+0000 7f67fffff640 1 -- 172.21.15.57:0/3959849770 >> v1:172.21.15.57:6790/0 conn(0x7f6818149dc0 legacy=0x7f68181e7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:08.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.636+0000 7f67fffff640 1 -- 172.21.15.57:0/3959849770 >> v1:172.21.15.57:6789/0 conn(0x7f68180a9930 legacy=0x7f68181e76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:08.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.636+0000 7f67fffff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:08.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.636+0000 7f67fffff640 1 monclient: found mon.b 2024-03-29T10:46:08.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.636+0000 7f67fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:08.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.636+0000 7f67fffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:46:08.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.636+0000 7f67fffff640 1 -- 172.21.15.57:0/3959849770 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681813c850 con 0x7f681814ff40 2024-03-29T10:46:08.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.636+0000 7f67fffff640 10 monclient: _finish_auth 0 2024-03-29T10:46:08.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.636+0000 7f67fffff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:08.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.636+0000 7f67fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:08.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.636+0000 7f681d93b640 5 monclient: authenticate success, global_id 4444 2024-03-29T10:46:08.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.636+0000 7f681d93b640 10 monclient: _renew_subs 2024-03-29T10:46:08.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.636+0000 7f681d93b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:46:08.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.636+0000 7f681d93b640 1 -- 172.21.15.57:0/3959849770 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681813bd90 con 0x7f681814ff40 2024-03-29T10:46:08.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.636+0000 7f681d93b640 10 client.4444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:08.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.636+0000 7f681d93b640 10 monclient: _renew_subs 2024-03-29T10:46:08.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.636+0000 7f681d93b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:46:08.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.636+0000 7f681d93b640 1 -- 172.21.15.57:0/3959849770 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f681813a970 con 0x7f681814ff40 2024-03-29T10:46:08.625 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.636+0000 7f681d93b640 1 librados: init done 2024-03-29T10:46:08.626 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.640+0000 7f67fffff640 1 -- 172.21.15.57:0/3959849770 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6810001cb0 con 0x7f681814ff40 2024-03-29T10:46:08.626 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.640+0000 7f67fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:08.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.640+0000 7f6814ff9640 4 set_mon_vals no callback set 2024-03-29T10:46:08.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.640+0000 7f681d93b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:08.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.641+0000 7f67fffff640 1 -- 172.21.15.57:0/3959849770 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f6810003b10 con 0x7f681814ff40 2024-03-29T10:46:08.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.641+0000 7f67fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:08.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.641+0000 7f67fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:08.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.641+0000 7f67fffff640 10 monclient: dump: 2024-03-29T10:46:08.627 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:08.627 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:08.627 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:08.627 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:08.627 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:08.627 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:08.627 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:08.627 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:08.628 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:08.628 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:08.628 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.641+0000 7f67fffff640 1 -- 172.21.15.57:0/3959849770 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f6810003d30 con 0x7f681814ff40 2024-03-29T10:46:08.628 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.642+0000 7f67fffff640 1 -- 172.21.15.57:0/3959849770 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f6810084bc0 con 0x7f681814ff40 2024-03-29T10:46:08.628 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.642+0000 7f67fffff640 10 client.4444.objecter ms_dispatch 0x7f6818000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:08.628 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.642+0000 7f67fffff640 3 client.4444.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:08.628 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.642+0000 7f67fffff640 3 client.4444.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:08.628 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.642+0000 7f67fffff640 20 client.4444.objecter dump_active .. 0 homeless 2024-03-29T10:46:08.628 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.642+0000 7f681d93b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:08.628 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.642+0000 7f681d93b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:46:08.628 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.642+0000 7f681d93b640 1 -- 172.21.15.57:0/3959849770 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67f8005080 con 0x7f681814ff40 2024-03-29T10:46:08.628 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.643+0000 7f6816ffd640 10 monclient: get_auth_request con 0x7f680406e160 auth_method 0 2024-03-29T10:46:08.629 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.647+0000 7f67fffff640 10 client.4444.objecter ms_handle_connect 0x7f680406e160 2024-03-29T10:46:08.630 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.648+0000 7f67fffff640 1 -- 172.21.15.57:0/3959849770 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6810050520 con 0x7f681814ff40 2024-03-29T10:46:08.630 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.648+0000 7f67fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:08.630 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:08.648+0000 7f67fffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:09.058 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:09.075+0000 7f681d93b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:09.058 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:09.075+0000 7f681d93b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:09.058 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:09.075+0000 7f681d93b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:46:09.058 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:09.075+0000 7f681d93b640 1 -- 172.21.15.57:0/3959849770 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67f8005330 con 0x7f681814ff40 2024-03-29T10:46:09.059 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:09.076+0000 7f67fffff640 1 -- 172.21.15.57:0/3959849770 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f6810052450 con 0x7f681814ff40 2024-03-29T10:46:09.059 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:09.076+0000 7f67fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:09.059 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:09.076+0000 7f67fffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:09.059 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:46:09.059 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:09.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:09.080+0000 7f681d93b640 10 librados: watch_flush enter 2024-03-29T10:46:09.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:09.080+0000 7f681d93b640 10 librados: watch_flush exit 2024-03-29T10:46:09.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:09.080+0000 7f681d93b640 20 client.4444.objecter shutdown clearing up homeless session... 2024-03-29T10:46:09.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:09.080+0000 7f681d93b640 10 client.4444.objecter successfully canceled tick 2024-03-29T10:46:09.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:09.080+0000 7f681d93b640 1 -- 172.21.15.57:0/3959849770 >> v1:172.21.15.179:6816/2698295936 conn(0x7f680406e160 legacy=0x7f6804070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:09.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:09.080+0000 7f681d93b640 10 monclient: shutdown 2024-03-29T10:46:09.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:09.080+0000 7f681d93b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:09.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:09.080+0000 7f681d93b640 1 -- 172.21.15.57:0/3959849770 >> v1:172.21.15.179:6789/0 conn(0x7f681814ff40 legacy=0x7f68181eeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:09.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:09.080+0000 7f681d93b640 1 -- 172.21.15.57:0/3959849770 shutdown_connections 2024-03-29T10:46:09.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:09.080+0000 7f681d93b640 1 -- 172.21.15.57:0/3959849770 >> 172.21.15.57:0/3959849770 conn(0x7f68180b35e0 msgr2=0x7f68180b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:09.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:09.080+0000 7f681d93b640 1 -- 172.21.15.57:0/3959849770 shutdown_connections 2024-03-29T10:46:09.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:09.080+0000 7f681d93b640 1 -- 172.21.15.57:0/3959849770 wait complete. 2024-03-29T10:46:09.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:09.081+0000 7f681d93b640 1 librados: shutdown 2024-03-29T10:46:09.079 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:15.885 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:16.034 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.052+0000 7f5b03b77640 10 monclient: get_monmap_and_config 2024-03-29T10:46:16.034 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.052+0000 7f5b03b77640 10 monclient: build_initial_monmap 2024-03-29T10:46:16.035 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.052+0000 7f5b03b77640 10 monclient: monmap: 2024-03-29T10:46:16.035 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:46:16.035 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:16.035 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:46:16.052959+0000 2024-03-29T10:46:16.035 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:46:16.052959+0000 2024-03-29T10:46:16.035 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:16.035 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:16.035 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:46:16.035 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:46:16.035 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:46:16.035 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:16.036 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.053+0000 7f5b03b77640 1 Processor -- start 2024-03-29T10:46:16.036 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.053+0000 7f5b03b77640 1 -- start start 2024-03-29T10:46:16.036 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.053+0000 7f5b03b77640 10 monclient: init 2024-03-29T10:46:16.036 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.054+0000 7f5b03b77640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:16.037 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.054+0000 7f5b03b77640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:46:16.037 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.054+0000 7f5b03b77640 10 monclient(hunting): picked mon.noname-c con 0x7f5afc1457f0 addr v1:172.21.15.179:6789/0 2024-03-29T10:46:16.037 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.054+0000 7f5b03b77640 10 monclient(hunting): picked mon.noname-b con 0x7f5afc13f650 addr v1:172.21.15.57:6790/0 2024-03-29T10:46:16.037 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.054+0000 7f5b03b77640 10 monclient(hunting): picked mon.noname-a con 0x7f5afc0a9d70 addr v1:172.21.15.57:6789/0 2024-03-29T10:46:16.037 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.054+0000 7f5b03b77640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5afc0aaef0 con 0x7f5afc0a9d70 2024-03-29T10:46:16.037 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.054+0000 7f5b03b77640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5afc145bd0 con 0x7f5afc1457f0 2024-03-29T10:46:16.037 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.054+0000 7f5b03b77640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5afc145eb0 con 0x7f5afc13f650 2024-03-29T10:46:16.037 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.054+0000 7f5b03b77640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:16.037 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.054+0000 7f5b03b77640 10 monclient(hunting): authenticate will time out at 1328.230327s 2024-03-29T10:46:16.037 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.054+0000 7f5b01112640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f5afc13f650 0x7f5afc0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:33392/0 (socket says 172.21.15.57:33392) 2024-03-29T10:46:16.037 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.054+0000 7f5b01112640 1 -- 172.21.15.57:0/1752967741 learned_addr learned my addr 172.21.15.57:0/1752967741 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:46:16.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 1 -- 172.21.15.57:0/1752967741 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275090076 0 0) 0x7f5afc145bd0 con 0x7f5afc1457f0 2024-03-29T10:46:16.040 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 10 monclient(hunting): _negotiate 2024-03-29T10:46:16.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:16.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:16.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 10 monclient(hunting): my global_id is 4447 2024-03-29T10:46:16.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 1 -- 172.21.15.57:0/1752967741 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ae8003520 con 0x7f5afc1457f0 2024-03-29T10:46:16.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 1 -- 172.21.15.57:0/1752967741 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 819137858 0 0) 0x7f5afc0aaef0 con 0x7f5afc0a9d70 2024-03-29T10:46:16.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 10 monclient(hunting): _negotiate 2024-03-29T10:46:16.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:16.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:16.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 10 monclient(hunting): my global_id is 4449 2024-03-29T10:46:16.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 1 -- 172.21.15.57:0/1752967741 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5afc145bd0 con 0x7f5afc0a9d70 2024-03-29T10:46:16.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 1 -- 172.21.15.57:0/1752967741 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549334485 0 0) 0x7f5afc145eb0 con 0x7f5afc13f650 2024-03-29T10:46:16.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 10 monclient(hunting): _negotiate 2024-03-29T10:46:16.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:16.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:16.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 10 monclient(hunting): my global_id is 4481 2024-03-29T10:46:16.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 1 -- 172.21.15.57:0/1752967741 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5afc0aaef0 con 0x7f5afc13f650 2024-03-29T10:46:16.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 1 -- 172.21.15.57:0/1752967741 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3055794005 0 0) 0x7f5ae8003520 con 0x7f5afc1457f0 2024-03-29T10:46:16.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 1 -- 172.21.15.57:0/1752967741 >> v1:172.21.15.57:6790/0 conn(0x7f5afc13f650 legacy=0x7f5afc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:16.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 1 -- 172.21.15.57:0/1752967741 >> v1:172.21.15.57:6789/0 conn(0x7f5afc0a9d70 legacy=0x7f5afc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:16.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 10 monclient: _finish_hunting 0 2024-03-29T10:46:16.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 1 monclient: found mon.noname-c 2024-03-29T10:46:16.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:16.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:46:16.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 1 -- 172.21.15.57:0/1752967741 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc146340 con 0x7f5afc1457f0 2024-03-29T10:46:16.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 10 monclient: _finish_auth 0 2024-03-29T10:46:16.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 10 monclient: _check_auth_tickets 2024-03-29T10:46:16.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:16.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 1 -- 172.21.15.57:0/1752967741 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f5aec001690 con 0x7f5afc1457f0 2024-03-29T10:46:16.042 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:16.043 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:16.043 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b00911640 10 monclient: dump: 2024-03-29T10:46:16.043 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:16.043 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:16.043 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:16.043 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:16.043 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:16.043 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:16.043 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:16.043 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:16.043 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:16.043 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:16.043 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b03b77640 5 monclient: authenticate success, global_id 4447 2024-03-29T10:46:16.043 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.055+0000 7f5b03b77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:16.043 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.056+0000 7f5b00911640 1 -- 172.21.15.57:0/1752967741 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5aec0017f0 con 0x7f5afc1457f0 2024-03-29T10:46:16.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.056+0000 7f5b00911640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:16.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.056+0000 7f5b00911640 1 -- 172.21.15.57:0/1752967741 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f5aec001c00 con 0x7f5afc1457f0 2024-03-29T10:46:16.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.056+0000 7f5b00911640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:16.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.056+0000 7f5b00911640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:16.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.056+0000 7f5b00911640 10 monclient: dump: 2024-03-29T10:46:16.044 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:16.044 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:16.044 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:16.044 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:16.044 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:16.044 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:16.044 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:16.044 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:16.044 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:16.044 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:16.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.056+0000 7f5b03b77640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:16.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.056+0000 7f5b03b77640 4 set_mon_vals no callback set 2024-03-29T10:46:16.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.057+0000 7f5b03b77640 10 monclient: shutdown 2024-03-29T10:46:16.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.057+0000 7f5b03b77640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:16.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.057+0000 7f5b03b77640 1 -- 172.21.15.57:0/1752967741 >> v1:172.21.15.179:6789/0 conn(0x7f5afc1457f0 legacy=0x7f5afc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:16.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.057+0000 7f5b03b77640 1 -- 172.21.15.57:0/1752967741 shutdown_connections 2024-03-29T10:46:16.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.057+0000 7f5b03b77640 1 -- 172.21.15.57:0/1752967741 >> 172.21.15.57:0/1752967741 conn(0x7f5afc066a00 msgr2=0x7f5afc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:16.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.057+0000 7f5b03b77640 1 -- 172.21.15.57:0/1752967741 shutdown_connections 2024-03-29T10:46:16.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.057+0000 7f5b03b77640 1 -- 172.21.15.57:0/1752967741 wait complete. 2024-03-29T10:46:16.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.058+0000 7f5b03b77640 10 monclient: build_initial_monmap 2024-03-29T10:46:16.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.058+0000 7f5b03b77640 10 monclient: monmap: 2024-03-29T10:46:16.045 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:46:16.045 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:16.045 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:46:16.045 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:46:16.045 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:16.046 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:16.046 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:46:16.046 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:46:16.046 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:46:16.046 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:16.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.058+0000 7f5b03b77640 1 librados: starting msgr at 2024-03-29T10:46:16.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.058+0000 7f5b03b77640 1 librados: starting objecter 2024-03-29T10:46:16.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.058+0000 7f5b03b77640 1 Processor -- start 2024-03-29T10:46:16.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.058+0000 7f5b03b77640 1 -- start start 2024-03-29T10:46:16.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.058+0000 7f5b03b77640 1 librados: setting wanted keys 2024-03-29T10:46:16.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.058+0000 7f5b03b77640 1 librados: calling monclient init 2024-03-29T10:46:16.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.058+0000 7f5b03b77640 10 monclient: init 2024-03-29T10:46:16.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.059+0000 7f5b03b77640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:16.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.059+0000 7f5b03b77640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:46:16.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.059+0000 7f5b03b77640 10 monclient(hunting): picked mon.noname-c con 0x7f5afc0a9d70 addr v1:172.21.15.57:6790/0 2024-03-29T10:46:16.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.059+0000 7f5b03b77640 10 monclient(hunting): picked mon.noname-b con 0x7f5afc13f650 addr v1:172.21.15.179:6789/0 2024-03-29T10:46:16.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.059+0000 7f5b03b77640 10 monclient(hunting): picked mon.noname-a con 0x7f5afc1457f0 addr v1:172.21.15.57:6789/0 2024-03-29T10:46:16.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.059+0000 7f5b03b77640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5afc0ac620 con 0x7f5afc1457f0 2024-03-29T10:46:16.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.059+0000 7f5b03b77640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5afc0ad550 con 0x7f5afc13f650 2024-03-29T10:46:16.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.059+0000 7f5b03b77640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5afc0ad7c0 con 0x7f5afc0a9d70 2024-03-29T10:46:16.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.059+0000 7f5b03b77640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:16.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.059+0000 7f5b03b77640 10 monclient(hunting): authenticate will time out at 1328.235729s 2024-03-29T10:46:16.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.059+0000 7f5b02114640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f5afc1457f0 0x7f5afc1d76f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:39860/0 (socket says 172.21.15.57:39860) 2024-03-29T10:46:16.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.059+0000 7f5b02114640 1 -- 172.21.15.57:0/434162419 learned_addr learned my addr 172.21.15.57:0/434162419 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:46:16.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.060+0000 7f5af27fc640 10 client.?.objecter ms_handle_connect 0x7f5afc0a9d70 2024-03-29T10:46:16.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.060+0000 7f5af27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:16.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.060+0000 7f5af27fc640 10 client.?.objecter ms_handle_connect 0x7f5afc1457f0 2024-03-29T10:46:16.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.060+0000 7f5af27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:16.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.060+0000 7f5af27fc640 10 client.?.objecter ms_handle_connect 0x7f5afc13f650 2024-03-29T10:46:16.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.060+0000 7f5af27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:16.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.060+0000 7f5af27fc640 1 -- 172.21.15.57:0/434162419 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 718993880 0 0) 0x7f5afc0ad7c0 con 0x7f5afc0a9d70 2024-03-29T10:46:16.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.060+0000 7f5af27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:46:16.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.060+0000 7f5af27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:16.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.060+0000 7f5af27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:16.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.060+0000 7f5af27fc640 10 monclient(hunting): my global_id is 4484 2024-03-29T10:46:16.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.060+0000 7f5af27fc640 1 -- 172.21.15.57:0/434162419 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ae0004eb0 con 0x7f5afc0a9d70 2024-03-29T10:46:16.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.060+0000 7f5af27fc640 1 -- 172.21.15.57:0/434162419 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22956840 0 0) 0x7f5afc0ac620 con 0x7f5afc1457f0 2024-03-29T10:46:16.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.060+0000 7f5af27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:46:16.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.060+0000 7f5af27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:16.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.060+0000 7f5af27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:16.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.060+0000 7f5af27fc640 10 monclient(hunting): my global_id is 4452 2024-03-29T10:46:16.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.060+0000 7f5af27fc640 1 -- 172.21.15.57:0/434162419 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5afc0ad7c0 con 0x7f5afc1457f0 2024-03-29T10:46:16.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 1 -- 172.21.15.57:0/434162419 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3537338852 0 0) 0x7f5afc0ad7c0 con 0x7f5afc1457f0 2024-03-29T10:46:16.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 1 -- 172.21.15.57:0/434162419 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5afc0ac620 con 0x7f5afc1457f0 2024-03-29T10:46:16.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 1 -- 172.21.15.57:0/434162419 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f5af4001c60 con 0x7f5afc1457f0 2024-03-29T10:46:16.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:16.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:16.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 10 monclient(hunting): dump: 2024-03-29T10:46:16.053 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:16.053 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:16.053 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:16.053 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:16.054 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:16.054 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:16.054 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:16.054 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:16.054 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:16.054 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:16.054 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:16.054 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 1 -- 172.21.15.57:0/434162419 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2091100983 0 0) 0x7f5ae0004eb0 con 0x7f5afc0a9d70 2024-03-29T10:46:16.054 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 1 -- 172.21.15.57:0/434162419 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5afc0ad7c0 con 0x7f5afc0a9d70 2024-03-29T10:46:16.054 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 1 -- 172.21.15.57:0/434162419 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f5aec001b40 con 0x7f5afc0a9d70 2024-03-29T10:46:16.054 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:16.054 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:16.054 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 10 monclient(hunting): dump: 2024-03-29T10:46:16.055 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:16.055 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:16.055 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:16.055 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:16.055 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:16.055 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:16.055 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:16.055 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:16.055 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:16.055 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:16.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 1 -- 172.21.15.57:0/434162419 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071959479 0 0) 0x7f5afc0ad550 con 0x7f5afc13f650 2024-03-29T10:46:16.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:46:16.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:16.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:16.056 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 10 monclient(hunting): my global_id is 4450 2024-03-29T10:46:16.056 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 1 -- 172.21.15.57:0/434162419 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ae0004eb0 con 0x7f5afc13f650 2024-03-29T10:46:16.056 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 1 -- 172.21.15.57:0/434162419 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4045642817 0 0) 0x7f5afc0ac620 con 0x7f5afc1457f0 2024-03-29T10:46:16.056 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 1 -- 172.21.15.57:0/434162419 >> v1:172.21.15.57:6790/0 conn(0x7f5afc0a9d70 legacy=0x7f5afc0abf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:16.056 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 1 -- 172.21.15.57:0/434162419 >> v1:172.21.15.179:6789/0 conn(0x7f5afc13f650 legacy=0x7f5afc138000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:16.056 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:46:16.056 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 1 monclient: found mon.a 2024-03-29T10:46:16.056 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:16.056 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:46:16.056 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 1 -- 172.21.15.57:0/434162419 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc0ab600 con 0x7f5afc1457f0 2024-03-29T10:46:16.056 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 10 monclient: _finish_auth 0 2024-03-29T10:46:16.056 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:46:16.056 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5af27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:16.056 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5b03b77640 5 monclient: authenticate success, global_id 4452 2024-03-29T10:46:16.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5b03b77640 10 monclient: _renew_subs 2024-03-29T10:46:16.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5b03b77640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:46:16.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5b03b77640 1 -- 172.21.15.57:0/434162419 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5afc138710 con 0x7f5afc1457f0 2024-03-29T10:46:16.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5b03b77640 10 client.4452.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:16.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5b03b77640 10 monclient: _renew_subs 2024-03-29T10:46:16.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5b03b77640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:46:16.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5b03b77640 1 -- 172.21.15.57:0/434162419 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5afc0b0f90 con 0x7f5afc1457f0 2024-03-29T10:46:16.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.061+0000 7f5b03b77640 1 librados: init done 2024-03-29T10:46:16.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.062+0000 7f5af27fc640 1 -- 172.21.15.57:0/434162419 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5af4003990 con 0x7f5afc1457f0 2024-03-29T10:46:16.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.062+0000 7f5af27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:16.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.062+0000 7f5af27fc640 1 -- 172.21.15.57:0/434162419 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f5af4003ce0 con 0x7f5afc1457f0 2024-03-29T10:46:16.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.062+0000 7f5af3fff640 4 set_mon_vals no callback set 2024-03-29T10:46:16.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.062+0000 7f5af27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:16.057 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.062+0000 7f5af27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:46:16.058 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.062+0000 7f5af27fc640 10 monclient: dump: 2024-03-29T10:46:16.058 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:16.058 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:16.058 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:16.058 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:16.058 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:16.058 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:16.058 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:16.058 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:16.058 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:16.058 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:16.058 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.063+0000 7f5b03b77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:16.058 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.063+0000 7f5b03b77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:16.058 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.063+0000 7f5b03b77640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:46:16.058 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.063+0000 7f5b03b77640 1 -- 172.21.15.57:0/434162419 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae4005080 con 0x7f5afc1457f0 2024-03-29T10:46:16.059 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.064+0000 7f5af27fc640 1 -- 172.21.15.57:0/434162419 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f5af4003e40 con 0x7f5afc1457f0 2024-03-29T10:46:16.059 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.065+0000 7f5af27fc640 1 -- 172.21.15.57:0/434162419 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f5af4084d50 con 0x7f5afc1457f0 2024-03-29T10:46:16.059 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.065+0000 7f5af27fc640 10 client.4452.objecter ms_dispatch 0x7f5afc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:16.059 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.065+0000 7f5af27fc640 3 client.4452.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:16.059 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.065+0000 7f5af27fc640 3 client.4452.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:16.059 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.065+0000 7f5af27fc640 20 client.4452.objecter dump_active .. 0 homeless 2024-03-29T10:46:16.059 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.065+0000 7f5b01913640 10 monclient: get_auth_request con 0x7f5ae006dfe0 auth_method 0 2024-03-29T10:46:16.059 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.065+0000 7f5af27fc640 10 client.4452.objecter ms_handle_connect 0x7f5ae006dfe0 2024-03-29T10:46:16.059 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.074+0000 7f5af27fc640 1 -- 172.21.15.57:0/434162419 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5af4050630 con 0x7f5afc1457f0 2024-03-29T10:46:16.059 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.074+0000 7f5af27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:16.060 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.074+0000 7f5af27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:16.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.465+0000 7f5b03b77640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:16.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.465+0000 7f5b03b77640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:16.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.465+0000 7f5b03b77640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:46:16.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.465+0000 7f5b03b77640 1 -- 172.21.15.57:0/434162419 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ae4005330 con 0x7f5afc1457f0 2024-03-29T10:46:16.449 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.467+0000 7f5af27fc640 1 -- 172.21.15.57:0/434162419 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f5af4052560 con 0x7f5afc1457f0 2024-03-29T10:46:16.449 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.467+0000 7f5af27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:16.449 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.467+0000 7f5af27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:16.450 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:46:16.450 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:16.453 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.471+0000 7f5b03b77640 10 librados: watch_flush enter 2024-03-29T10:46:16.453 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.471+0000 7f5b03b77640 10 librados: watch_flush exit 2024-03-29T10:46:16.453 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.471+0000 7f5b03b77640 20 client.4452.objecter shutdown clearing up homeless session... 2024-03-29T10:46:16.453 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.471+0000 7f5b03b77640 10 client.4452.objecter successfully canceled tick 2024-03-29T10:46:16.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.471+0000 7f5b03b77640 1 -- 172.21.15.57:0/434162419 >> v1:172.21.15.179:6816/2698295936 conn(0x7f5ae006dfe0 legacy=0x7f5ae00704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:16.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.471+0000 7f5b03b77640 10 monclient: shutdown 2024-03-29T10:46:16.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.471+0000 7f5b03b77640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:16.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.471+0000 7f5b03b77640 1 -- 172.21.15.57:0/434162419 >> v1:172.21.15.57:6789/0 conn(0x7f5afc1457f0 legacy=0x7f5afc1d76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:16.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.471+0000 7f5b03b77640 1 -- 172.21.15.57:0/434162419 shutdown_connections 2024-03-29T10:46:16.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.471+0000 7f5b03b77640 1 -- 172.21.15.57:0/434162419 >> 172.21.15.57:0/434162419 conn(0x7f5afc066a00 msgr2=0x7f5afc0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:16.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.471+0000 7f5b03b77640 1 -- 172.21.15.57:0/434162419 shutdown_connections 2024-03-29T10:46:16.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.471+0000 7f5b03b77640 1 -- 172.21.15.57:0/434162419 wait complete. 2024-03-29T10:46:16.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:16.471+0000 7f5b03b77640 1 librados: shutdown 2024-03-29T10:46:16.469 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:23.277 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:23.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.431+0000 7f4b67332640 10 monclient: get_monmap_and_config 2024-03-29T10:46:23.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.432+0000 7f4b67332640 10 monclient: build_initial_monmap 2024-03-29T10:46:23.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.432+0000 7f4b67332640 10 monclient: monmap: 2024-03-29T10:46:23.415 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:46:23.415 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:23.415 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:46:23.432729+0000 2024-03-29T10:46:23.415 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:46:23.432729+0000 2024-03-29T10:46:23.415 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:23.415 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:23.415 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:46:23.415 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:46:23.415 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:46:23.416 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:23.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.432+0000 7f4b67332640 1 Processor -- start 2024-03-29T10:46:23.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.432+0000 7f4b67332640 1 -- start start 2024-03-29T10:46:23.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.432+0000 7f4b67332640 10 monclient: init 2024-03-29T10:46:23.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.433+0000 7f4b67332640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:23.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.433+0000 7f4b67332640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:46:23.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.433+0000 7f4b67332640 10 monclient(hunting): picked mon.noname-b con 0x7f4b600b0860 addr v1:172.21.15.57:6790/0 2024-03-29T10:46:23.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.433+0000 7f4b67332640 10 monclient(hunting): picked mon.noname-a con 0x7f4b60072450 addr v1:172.21.15.57:6789/0 2024-03-29T10:46:23.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.434+0000 7f4b67332640 10 monclient(hunting): picked mon.noname-c con 0x7f4b6013ee40 addr v1:172.21.15.179:6789/0 2024-03-29T10:46:23.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.434+0000 7f4b67332640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b600ad840 con 0x7f4b60072450 2024-03-29T10:46:23.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.434+0000 7f4b67332640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b6013c830 con 0x7f4b6013ee40 2024-03-29T10:46:23.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.434+0000 7f4b67332640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b6013c9c0 con 0x7f4b600b0860 2024-03-29T10:46:23.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.434+0000 7f4b67332640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:23.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.434+0000 7f4b67332640 10 monclient(hunting): authenticate will time out at 1335.610272s 2024-03-29T10:46:23.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.434+0000 7f4b658cf640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f4b6013ee40 0x7f4b6013f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:34286/0 (socket says 172.21.15.57:34286) 2024-03-29T10:46:23.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.434+0000 7f4b658cf640 1 -- 172.21.15.57:0/2484546702 learned_addr learned my addr 172.21.15.57:0/2484546702 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:46:23.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.434+0000 7f4b57fff640 1 -- 172.21.15.57:0/2484546702 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551920018 0 0) 0x7f4b6013c830 con 0x7f4b6013ee40 2024-03-29T10:46:23.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.435+0000 7f4b57fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:23.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.435+0000 7f4b57fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:23.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.435+0000 7f4b57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:23.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.435+0000 7f4b57fff640 10 monclient(hunting): my global_id is 4453 2024-03-29T10:46:23.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.435+0000 7f4b57fff640 1 -- 172.21.15.57:0/2484546702 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b4c003520 con 0x7f4b6013ee40 2024-03-29T10:46:23.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.435+0000 7f4b57fff640 1 -- 172.21.15.57:0/2484546702 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282230177 0 0) 0x7f4b6013c9c0 con 0x7f4b600b0860 2024-03-29T10:46:23.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.435+0000 7f4b57fff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:23.419 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.435+0000 7f4b57fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:23.419 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.435+0000 7f4b57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:23.419 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.435+0000 7f4b57fff640 10 monclient(hunting): my global_id is 4487 2024-03-29T10:46:23.419 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.435+0000 7f4b57fff640 1 -- 172.21.15.57:0/2484546702 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b6013c830 con 0x7f4b600b0860 2024-03-29T10:46:23.419 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.435+0000 7f4b57fff640 1 -- 172.21.15.57:0/2484546702 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2541256237 0 0) 0x7f4b4c003520 con 0x7f4b6013ee40 2024-03-29T10:46:23.419 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.435+0000 7f4b57fff640 1 -- 172.21.15.57:0/2484546702 >> v1:172.21.15.57:6790/0 conn(0x7f4b600b0860 legacy=0x7f4b600acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:23.419 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.435+0000 7f4b57fff640 1 -- 172.21.15.57:0/2484546702 >> v1:172.21.15.57:6789/0 conn(0x7f4b60072450 legacy=0x7f4b6013c120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:23.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.437+0000 7f4b57fff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:23.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.437+0000 7f4b57fff640 1 monclient: found mon.noname-c 2024-03-29T10:46:23.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.437+0000 7f4b57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:23.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.437+0000 7f4b57fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:46:23.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.437+0000 7f4b57fff640 1 -- 172.21.15.57:0/2484546702 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6013f970 con 0x7f4b6013ee40 2024-03-29T10:46:23.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.437+0000 7f4b57fff640 10 monclient: _finish_auth 0 2024-03-29T10:46:23.421 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.437+0000 7f4b57fff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:23.421 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.437+0000 7f4b57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:23.421 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.437+0000 7f4b57fff640 1 -- 172.21.15.57:0/2484546702 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f4b58001a10 con 0x7f4b6013ee40 2024-03-29T10:46:23.421 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.437+0000 7f4b57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:23.421 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.437+0000 7f4b57fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:23.421 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.437+0000 7f4b57fff640 10 monclient: dump: 2024-03-29T10:46:23.421 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:23.421 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:23.421 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:23.421 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:23.421 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:23.421 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:23.421 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:23.421 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:23.422 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:23.422 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:23.422 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.437+0000 7f4b67332640 5 monclient: authenticate success, global_id 4453 2024-03-29T10:46:23.422 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.437+0000 7f4b67332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:23.422 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.437+0000 7f4b57fff640 1 -- 172.21.15.57:0/2484546702 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b58001b70 con 0x7f4b6013ee40 2024-03-29T10:46:23.422 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.437+0000 7f4b57fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:23.422 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.437+0000 7f4b57fff640 1 -- 172.21.15.57:0/2484546702 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f4b58001fc0 con 0x7f4b6013ee40 2024-03-29T10:46:23.422 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.437+0000 7f4b57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:23.422 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.437+0000 7f4b57fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:23.422 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.437+0000 7f4b57fff640 10 monclient: dump: 2024-03-29T10:46:23.422 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:23.422 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:23.422 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:23.422 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:23.422 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:23.423 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:23.423 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:23.423 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:23.423 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:23.423 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:23.423 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.437+0000 7f4b67332640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:23.423 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.437+0000 7f4b67332640 4 set_mon_vals no callback set 2024-03-29T10:46:23.425 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.438+0000 7f4b67332640 10 monclient: shutdown 2024-03-29T10:46:23.425 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.438+0000 7f4b67332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:23.425 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.438+0000 7f4b67332640 1 -- 172.21.15.57:0/2484546702 >> v1:172.21.15.179:6789/0 conn(0x7f4b6013ee40 legacy=0x7f4b6013f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:23.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.439+0000 7f4b67332640 1 -- 172.21.15.57:0/2484546702 shutdown_connections 2024-03-29T10:46:23.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.439+0000 7f4b67332640 1 -- 172.21.15.57:0/2484546702 >> 172.21.15.57:0/2484546702 conn(0x7f4b6006b660 msgr2=0x7f4b6006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:23.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.439+0000 7f4b67332640 1 -- 172.21.15.57:0/2484546702 shutdown_connections 2024-03-29T10:46:23.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.439+0000 7f4b67332640 1 -- 172.21.15.57:0/2484546702 wait complete. 2024-03-29T10:46:23.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.440+0000 7f4b67332640 10 monclient: build_initial_monmap 2024-03-29T10:46:23.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.440+0000 7f4b67332640 10 monclient: monmap: 2024-03-29T10:46:23.426 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:46:23.426 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:23.426 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:46:23.426 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:46:23.426 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:23.426 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:23.426 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:46:23.426 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:46:23.427 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:46:23.427 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:23.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.440+0000 7f4b67332640 1 librados: starting msgr at 2024-03-29T10:46:23.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.440+0000 7f4b67332640 1 librados: starting objecter 2024-03-29T10:46:23.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.440+0000 7f4b67332640 1 Processor -- start 2024-03-29T10:46:23.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.440+0000 7f4b67332640 1 -- start start 2024-03-29T10:46:23.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.440+0000 7f4b67332640 1 librados: setting wanted keys 2024-03-29T10:46:23.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.440+0000 7f4b67332640 1 librados: calling monclient init 2024-03-29T10:46:23.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.440+0000 7f4b67332640 10 monclient: init 2024-03-29T10:46:23.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.441+0000 7f4b67332640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:23.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.441+0000 7f4b67332640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:46:23.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.441+0000 7f4b67332640 10 monclient(hunting): picked mon.noname-b con 0x7f4b60072450 addr v1:172.21.15.179:6789/0 2024-03-29T10:46:23.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.441+0000 7f4b67332640 10 monclient(hunting): picked mon.noname-a con 0x7f4b600b0860 addr v1:172.21.15.57:6789/0 2024-03-29T10:46:23.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.441+0000 7f4b67332640 10 monclient(hunting): picked mon.noname-c con 0x7f4b6013ee40 addr v1:172.21.15.57:6790/0 2024-03-29T10:46:23.428 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.441+0000 7f4b67332640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b600b43f0 con 0x7f4b600b0860 2024-03-29T10:46:23.428 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.441+0000 7f4b67332640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b600b3520 con 0x7f4b60072450 2024-03-29T10:46:23.428 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.441+0000 7f4b67332640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b600b3100 con 0x7f4b6013ee40 2024-03-29T10:46:23.428 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.441+0000 7f4b67332640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:23.428 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.441+0000 7f4b67332640 10 monclient(hunting): authenticate will time out at 1335.617927s 2024-03-29T10:46:23.428 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.442+0000 7f4b658cf640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f4b6013ee40 0x7f4b601dd5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:54352/0 (socket says 172.21.15.57:54352) 2024-03-29T10:46:23.428 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.442+0000 7f4b658cf640 1 -- 172.21.15.57:0/13961310 learned_addr learned my addr 172.21.15.57:0/13961310 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:46:23.428 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.442+0000 7f4b55ffb640 10 client.?.objecter ms_handle_connect 0x7f4b60072450 2024-03-29T10:46:23.428 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.442+0000 7f4b55ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:23.428 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.442+0000 7f4b55ffb640 10 client.?.objecter ms_handle_connect 0x7f4b6013ee40 2024-03-29T10:46:23.428 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.442+0000 7f4b55ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:23.428 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.442+0000 7f4b55ffb640 1 -- 172.21.15.57:0/13961310 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424419950 0 0) 0x7f4b600b3520 con 0x7f4b60072450 2024-03-29T10:46:23.428 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.442+0000 7f4b55ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:23.428 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.442+0000 7f4b55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:23.429 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.442+0000 7f4b55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:23.429 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.442+0000 7f4b55ffb640 10 monclient(hunting): my global_id is 4456 2024-03-29T10:46:23.429 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.442+0000 7f4b55ffb640 1 -- 172.21.15.57:0/13961310 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b44004eb0 con 0x7f4b60072450 2024-03-29T10:46:23.429 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.443+0000 7f4b55ffb640 1 -- 172.21.15.57:0/13961310 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402086393 0 0) 0x7f4b600b3100 con 0x7f4b6013ee40 2024-03-29T10:46:23.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.443+0000 7f4b55ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:23.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.443+0000 7f4b55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:23.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.443+0000 7f4b55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:23.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.443+0000 7f4b55ffb640 10 monclient(hunting): my global_id is 4490 2024-03-29T10:46:23.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.443+0000 7f4b55ffb640 1 -- 172.21.15.57:0/13961310 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b440051d0 con 0x7f4b6013ee40 2024-03-29T10:46:23.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.443+0000 7f4b55ffb640 1 -- 172.21.15.57:0/13961310 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 42472961 0 0) 0x7f4b44004eb0 con 0x7f4b60072450 2024-03-29T10:46:23.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.443+0000 7f4b55ffb640 1 -- 172.21.15.57:0/13961310 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b600b3100 con 0x7f4b60072450 2024-03-29T10:46:23.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.443+0000 7f4b55ffb640 1 -- 172.21.15.57:0/13961310 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f4b5c001a10 con 0x7f4b60072450 2024-03-29T10:46:23.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.443+0000 7f4b55ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:23.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.443+0000 7f4b55ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:23.430 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.443+0000 7f4b55ffb640 10 monclient(hunting): dump: 2024-03-29T10:46:23.430 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:23.430 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:23.431 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:23.431 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:23.431 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:23.431 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:23.431 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:23.431 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:23.431 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:23.431 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:23.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.443+0000 7f4b55ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:23.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.443+0000 7f4b55ffb640 1 -- 172.21.15.57:0/13961310 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3426291328 0 0) 0x7f4b440051d0 con 0x7f4b6013ee40 2024-03-29T10:46:23.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.443+0000 7f4b55ffb640 1 -- 172.21.15.57:0/13961310 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b44005b60 con 0x7f4b6013ee40 2024-03-29T10:46:23.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.443+0000 7f4b55ffb640 1 -- 172.21.15.57:0/13961310 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f4b58001f40 con 0x7f4b6013ee40 2024-03-29T10:46:23.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.443+0000 7f4b55ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:23.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.443+0000 7f4b55ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:23.432 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.443+0000 7f4b55ffb640 10 monclient(hunting): dump: 2024-03-29T10:46:23.432 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:23.432 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:23.432 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:23.432 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:23.432 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:23.432 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:23.432 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:23.432 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:23.432 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:23.432 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:23.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.443+0000 7f4b55ffb640 1 -- 172.21.15.57:0/13961310 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2571765029 0 0) 0x7f4b600b3100 con 0x7f4b60072450 2024-03-29T10:46:23.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.443+0000 7f4b55ffb640 1 -- 172.21.15.57:0/13961310 >> v1:172.21.15.57:6790/0 conn(0x7f4b6013ee40 legacy=0x7f4b601dd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:23.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.443+0000 7f4b55ffb640 1 -- 172.21.15.57:0/13961310 >> v1:172.21.15.57:6789/0 conn(0x7f4b600b0860 legacy=0x7f4b600b1440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:46:23.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.444+0000 7f4b55ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:46:23.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.444+0000 7f4b55ffb640 1 monclient: found mon.b 2024-03-29T10:46:23.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.444+0000 7f4b55ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:23.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.444+0000 7f4b55ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:46:23.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.444+0000 7f4b55ffb640 1 -- 172.21.15.57:0/13961310 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b600b37c0 con 0x7f4b60072450 2024-03-29T10:46:23.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.444+0000 7f4b55ffb640 10 monclient: _finish_auth 0 2024-03-29T10:46:23.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.444+0000 7f4b55ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:46:23.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.444+0000 7f4b55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:23.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.444+0000 7f4b67332640 5 monclient: authenticate success, global_id 4456 2024-03-29T10:46:23.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.445+0000 7f4b67332640 10 monclient: _renew_subs 2024-03-29T10:46:23.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.445+0000 7f4b67332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:46:23.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.445+0000 7f4b67332640 1 -- 172.21.15.57:0/13961310 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b600b1b50 con 0x7f4b60072450 2024-03-29T10:46:23.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.445+0000 7f4b55ffb640 1 -- 172.21.15.57:0/13961310 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b5c0035f0 con 0x7f4b60072450 2024-03-29T10:46:23.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.445+0000 7f4b55ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:23.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.445+0000 7f4b55ffb640 1 -- 172.21.15.57:0/13961310 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f4b5c001a10 con 0x7f4b60072450 2024-03-29T10:46:23.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.445+0000 7f4b55ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:23.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.445+0000 7f4b55ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:23.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.445+0000 7f4b55ffb640 10 monclient: dump: 2024-03-29T10:46:23.435 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:23.435 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:23.436 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:23.436 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:23.436 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:23.436 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:23.436 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:23.436 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:23.436 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:23.436 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:23.436 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.445+0000 7f4b577fe640 4 set_mon_vals no callback set 2024-03-29T10:46:23.436 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.445+0000 7f4b67332640 10 client.4456.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:23.436 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.445+0000 7f4b67332640 10 monclient: _renew_subs 2024-03-29T10:46:23.436 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.445+0000 7f4b67332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:46:23.436 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.445+0000 7f4b67332640 1 -- 172.21.15.57:0/13961310 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b601decf0 con 0x7f4b60072450 2024-03-29T10:46:23.436 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.445+0000 7f4b67332640 1 librados: init done 2024-03-29T10:46:23.436 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.447+0000 7f4b67332640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:23.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.447+0000 7f4b67332640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:23.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.447+0000 7f4b67332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:46:23.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.447+0000 7f4b67332640 1 -- 172.21.15.57:0/13961310 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b600b3a00 con 0x7f4b60072450 2024-03-29T10:46:23.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.447+0000 7f4b55ffb640 1 -- 172.21.15.57:0/13961310 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f4b5c0195d0 con 0x7f4b60072450 2024-03-29T10:46:23.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.448+0000 7f4b55ffb640 1 -- 172.21.15.57:0/13961310 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f4b5c0834a0 con 0x7f4b60072450 2024-03-29T10:46:23.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.448+0000 7f4b55ffb640 10 client.4456.objecter ms_dispatch 0x7f4b60000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:23.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.448+0000 7f4b55ffb640 3 client.4456.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:23.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.448+0000 7f4b55ffb640 3 client.4456.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:23.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.448+0000 7f4b55ffb640 20 client.4456.objecter dump_active .. 0 homeless 2024-03-29T10:46:23.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.448+0000 7f4b648cd640 10 monclient: get_auth_request con 0x7f4b4406e3c0 auth_method 0 2024-03-29T10:46:23.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.448+0000 7f4b55ffb640 10 client.4456.objecter ms_handle_connect 0x7f4b4406e3c0 2024-03-29T10:46:23.438 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.456+0000 7f4b55ffb640 1 -- 172.21.15.57:0/13961310 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b600b3a00 con 0x7f4b60072450 2024-03-29T10:46:23.438 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.456+0000 7f4b55ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:23.438 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.456+0000 7f4b55ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:23.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.781+0000 7f4b67332640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:23.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.781+0000 7f4b67332640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:23.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.781+0000 7f4b67332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:46:23.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.781+0000 7f4b67332640 1 -- 172.21.15.57:0/13961310 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b600b3a00 con 0x7f4b60072450 2024-03-29T10:46:23.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.782+0000 7f4b55ffb640 1 -- 172.21.15.57:0/13961310 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f4b600b3a00 con 0x7f4b60072450 2024-03-29T10:46:23.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.782+0000 7f4b55ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:23.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.782+0000 7f4b55ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:23.765 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:46:23.765 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:23.768 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.785+0000 7f4b67332640 10 librados: watch_flush enter 2024-03-29T10:46:23.768 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.785+0000 7f4b67332640 10 librados: watch_flush exit 2024-03-29T10:46:23.768 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.785+0000 7f4b67332640 20 client.4456.objecter shutdown clearing up homeless session... 2024-03-29T10:46:23.768 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.785+0000 7f4b67332640 10 client.4456.objecter successfully canceled tick 2024-03-29T10:46:23.768 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.785+0000 7f4b67332640 1 -- 172.21.15.57:0/13961310 >> v1:172.21.15.179:6816/2698295936 conn(0x7f4b4406e3c0 legacy=0x7f4b44070880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:23.768 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.785+0000 7f4b67332640 10 monclient: shutdown 2024-03-29T10:46:23.768 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.785+0000 7f4b67332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:23.769 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.785+0000 7f4b67332640 1 -- 172.21.15.57:0/13961310 >> v1:172.21.15.179:6789/0 conn(0x7f4b60072450 legacy=0x7f4b600b3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:23.769 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.785+0000 7f4b67332640 1 -- 172.21.15.57:0/13961310 shutdown_connections 2024-03-29T10:46:23.769 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.785+0000 7f4b67332640 1 -- 172.21.15.57:0/13961310 >> 172.21.15.57:0/13961310 conn(0x7f4b6006b660 msgr2=0x7f4b600a9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:23.769 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.785+0000 7f4b67332640 1 -- 172.21.15.57:0/13961310 shutdown_connections 2024-03-29T10:46:23.769 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.786+0000 7f4b67332640 1 -- 172.21.15.57:0/13961310 wait complete. 2024-03-29T10:46:23.769 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:23.786+0000 7f4b67332640 1 librados: shutdown 2024-03-29T10:46:23.784 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:30.591 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:30.725 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.743+0000 7f8e29a4e640 10 monclient: get_monmap_and_config 2024-03-29T10:46:30.725 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.743+0000 7f8e29a4e640 10 monclient: build_initial_monmap 2024-03-29T10:46:30.725 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.743+0000 7f8e29a4e640 10 monclient: monmap: 2024-03-29T10:46:30.725 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:46:30.725 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:30.726 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:46:30.743843+0000 2024-03-29T10:46:30.726 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:46:30.743843+0000 2024-03-29T10:46:30.726 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:30.726 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:30.726 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:46:30.726 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:46:30.727 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:46:30.727 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:30.732 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.743+0000 7f8e29a4e640 1 Processor -- start 2024-03-29T10:46:30.732 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.743+0000 7f8e29a4e640 1 -- start start 2024-03-29T10:46:30.732 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.743+0000 7f8e29a4e640 10 monclient: init 2024-03-29T10:46:30.732 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.744+0000 7f8e29a4e640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:30.732 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.744+0000 7f8e29a4e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:46:30.732 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.744+0000 7f8e29a4e640 10 monclient(hunting): picked mon.noname-c con 0x7f8e241457f0 addr v1:172.21.15.179:6789/0 2024-03-29T10:46:30.733 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.744+0000 7f8e29a4e640 10 monclient(hunting): picked mon.noname-a con 0x7f8e2413f650 addr v1:172.21.15.57:6789/0 2024-03-29T10:46:30.733 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.744+0000 7f8e29a4e640 10 monclient(hunting): picked mon.noname-b con 0x7f8e240a9d70 addr v1:172.21.15.57:6790/0 2024-03-29T10:46:30.733 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.744+0000 7f8e29a4e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e240aaef0 con 0x7f8e2413f650 2024-03-29T10:46:30.733 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.744+0000 7f8e29a4e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e24145bd0 con 0x7f8e241457f0 2024-03-29T10:46:30.733 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.744+0000 7f8e29a4e640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e24145eb0 con 0x7f8e240a9d70 2024-03-29T10:46:30.733 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.744+0000 7f8e29a4e640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:30.733 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.744+0000 7f8e29a4e640 10 monclient(hunting): authenticate will time out at 1342.920748s 2024-03-29T10:46:30.733 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.744+0000 7f8e22ffd640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f8e241457f0 0x7f8e2413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:34106/0 (socket says 172.21.15.57:34106) 2024-03-29T10:46:30.733 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.744+0000 7f8e22ffd640 1 -- 172.21.15.57:0/1389123297 learned_addr learned my addr 172.21.15.57:0/1389123297 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:46:30.733 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.745+0000 7f8e21ffb640 1 -- 172.21.15.57:0/1389123297 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655628437 0 0) 0x7f8e24145eb0 con 0x7f8e240a9d70 2024-03-29T10:46:30.734 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.745+0000 7f8e21ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:30.734 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.745+0000 7f8e21ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:30.734 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.745+0000 7f8e21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:30.734 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.745+0000 7f8e21ffb640 10 monclient(hunting): my global_id is 4493 2024-03-29T10:46:30.734 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.745+0000 7f8e21ffb640 1 -- 172.21.15.57:0/1389123297 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e0c003520 con 0x7f8e240a9d70 2024-03-29T10:46:30.734 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.745+0000 7f8e21ffb640 1 -- 172.21.15.57:0/1389123297 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 486357017 0 0) 0x7f8e24145bd0 con 0x7f8e241457f0 2024-03-29T10:46:30.734 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.745+0000 7f8e21ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:30.734 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.745+0000 7f8e21ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:30.734 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.745+0000 7f8e21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:30.734 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.745+0000 7f8e21ffb640 10 monclient(hunting): my global_id is 4459 2024-03-29T10:46:30.734 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.745+0000 7f8e21ffb640 1 -- 172.21.15.57:0/1389123297 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e24145eb0 con 0x7f8e241457f0 2024-03-29T10:46:30.735 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.745+0000 7f8e21ffb640 1 -- 172.21.15.57:0/1389123297 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1806069563 0 0) 0x7f8e0c003520 con 0x7f8e240a9d70 2024-03-29T10:46:30.735 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.745+0000 7f8e21ffb640 1 -- 172.21.15.57:0/1389123297 >> v1:172.21.15.179:6789/0 conn(0x7f8e241457f0 legacy=0x7f8e2413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:30.735 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.745+0000 7f8e21ffb640 1 -- 172.21.15.57:0/1389123297 >> v1:172.21.15.57:6789/0 conn(0x7f8e2413f650 legacy=0x7f8e240b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:30.735 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.748+0000 7f8e21ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:46:30.735 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.748+0000 7f8e21ffb640 1 monclient: found mon.noname-b 2024-03-29T10:46:30.735 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.748+0000 7f8e21ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:30.735 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.748+0000 7f8e21ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:46:30.736 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.748+0000 7f8e21ffb640 1 -- 172.21.15.57:0/1389123297 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e24146340 con 0x7f8e240a9d70 2024-03-29T10:46:30.736 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.748+0000 7f8e21ffb640 10 monclient: _finish_auth 0 2024-03-29T10:46:30.736 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.748+0000 7f8e21ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:46:30.736 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.748+0000 7f8e21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:30.736 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.748+0000 7f8e21ffb640 1 -- 172.21.15.57:0/1389123297 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f8e1c001a10 con 0x7f8e240a9d70 2024-03-29T10:46:30.736 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.748+0000 7f8e21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:30.737 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.748+0000 7f8e21ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:30.737 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.748+0000 7f8e21ffb640 10 monclient: dump: 2024-03-29T10:46:30.737 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:30.737 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:30.738 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:30.738 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:30.738 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:30.738 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:30.738 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:30.738 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:30.738 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:30.739 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:30.739 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.748+0000 7f8e29a4e640 5 monclient: authenticate success, global_id 4493 2024-03-29T10:46:30.739 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.748+0000 7f8e29a4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:30.739 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.749+0000 7f8e21ffb640 1 -- 172.21.15.57:0/1389123297 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e1c001b70 con 0x7f8e240a9d70 2024-03-29T10:46:30.739 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.749+0000 7f8e21ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:30.739 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.749+0000 7f8e21ffb640 1 -- 172.21.15.57:0/1389123297 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f8e1c001fc0 con 0x7f8e240a9d70 2024-03-29T10:46:30.740 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.749+0000 7f8e21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:30.740 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.749+0000 7f8e21ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:30.740 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.749+0000 7f8e21ffb640 10 monclient: dump: 2024-03-29T10:46:30.740 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:30.740 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:30.740 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:30.740 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:30.740 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:30.741 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:30.741 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:30.741 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:30.742 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:30.742 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:30.742 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.749+0000 7f8e29a4e640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:30.742 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.749+0000 7f8e29a4e640 4 set_mon_vals no callback set 2024-03-29T10:46:30.742 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.750+0000 7f8e29a4e640 10 monclient: shutdown 2024-03-29T10:46:30.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.750+0000 7f8e29a4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:30.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.750+0000 7f8e29a4e640 1 -- 172.21.15.57:0/1389123297 >> v1:172.21.15.57:6790/0 conn(0x7f8e240a9d70 legacy=0x7f8e2414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:30.749 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.750+0000 7f8e29a4e640 1 -- 172.21.15.57:0/1389123297 shutdown_connections 2024-03-29T10:46:30.750 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.750+0000 7f8e29a4e640 1 -- 172.21.15.57:0/1389123297 >> 172.21.15.57:0/1389123297 conn(0x7f8e24066a00 msgr2=0x7f8e24066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:30.750 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.751+0000 7f8e29a4e640 1 -- 172.21.15.57:0/1389123297 shutdown_connections 2024-03-29T10:46:30.750 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.751+0000 7f8e29a4e640 1 -- 172.21.15.57:0/1389123297 wait complete. 2024-03-29T10:46:30.750 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.752+0000 7f8e29a4e640 10 monclient: build_initial_monmap 2024-03-29T10:46:30.750 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.752+0000 7f8e29a4e640 10 monclient: monmap: 2024-03-29T10:46:30.750 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:46:30.750 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:30.750 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:46:30.750 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:46:30.750 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:30.750 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:30.750 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:46:30.750 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:46:30.751 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:46:30.751 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:30.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.753+0000 7f8e29a4e640 1 librados: starting msgr at 2024-03-29T10:46:30.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.753+0000 7f8e29a4e640 1 librados: starting objecter 2024-03-29T10:46:30.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.753+0000 7f8e29a4e640 1 Processor -- start 2024-03-29T10:46:30.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.753+0000 7f8e29a4e640 1 -- start start 2024-03-29T10:46:30.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.753+0000 7f8e29a4e640 1 librados: setting wanted keys 2024-03-29T10:46:30.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.753+0000 7f8e29a4e640 1 librados: calling monclient init 2024-03-29T10:46:30.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.753+0000 7f8e29a4e640 10 monclient: init 2024-03-29T10:46:30.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.754+0000 7f8e29a4e640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:30.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.754+0000 7f8e29a4e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:46:30.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.754+0000 7f8e29a4e640 10 monclient(hunting): picked mon.noname-a con 0x7f8e240a9d70 addr v1:172.21.15.57:6789/0 2024-03-29T10:46:30.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.755+0000 7f8e29a4e640 10 monclient(hunting): picked mon.noname-c con 0x7f8e2413f650 addr v1:172.21.15.57:6790/0 2024-03-29T10:46:30.751 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.755+0000 7f8e29a4e640 10 monclient(hunting): picked mon.noname-b con 0x7f8e241457f0 addr v1:172.21.15.179:6789/0 2024-03-29T10:46:30.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.755+0000 7f8e29a4e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e24141aa0 con 0x7f8e240a9d70 2024-03-29T10:46:30.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.755+0000 7f8e29a4e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e24142e70 con 0x7f8e241457f0 2024-03-29T10:46:30.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.755+0000 7f8e29a4e640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e241422a0 con 0x7f8e2413f650 2024-03-29T10:46:30.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.755+0000 7f8e29a4e640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:30.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.755+0000 7f8e29a4e640 10 monclient(hunting): authenticate will time out at 1342.931409s 2024-03-29T10:46:30.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.756+0000 7f8e227fc640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f8e2413f650 0x7f8e241dedd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:33390/0 (socket says 172.21.15.57:33390) 2024-03-29T10:46:30.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.756+0000 7f8e227fc640 1 -- 172.21.15.57:0/1905035311 learned_addr learned my addr 172.21.15.57:0/1905035311 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:46:30.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.756+0000 7f8e0bfff640 10 client.?.objecter ms_handle_connect 0x7f8e2413f650 2024-03-29T10:46:30.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.756+0000 7f8e0bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:30.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.756+0000 7f8e0bfff640 1 -- 172.21.15.57:0/1905035311 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3778950951 0 0) 0x7f8e241422a0 con 0x7f8e2413f650 2024-03-29T10:46:30.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.756+0000 7f8e0bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:30.752 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.756+0000 7f8e0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:30.753 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.756+0000 7f8e0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:30.753 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.756+0000 7f8e0bfff640 10 monclient(hunting): my global_id is 4496 2024-03-29T10:46:30.753 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.756+0000 7f8e0bfff640 1 -- 172.21.15.57:0/1905035311 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e10004eb0 con 0x7f8e2413f650 2024-03-29T10:46:30.753 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.756+0000 7f8e0bfff640 10 client.?.objecter ms_handle_connect 0x7f8e241457f0 2024-03-29T10:46:30.753 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.756+0000 7f8e0bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:30.753 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.757+0000 7f8e0bfff640 1 -- 172.21.15.57:0/1905035311 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2238489232 0 0) 0x7f8e10004eb0 con 0x7f8e2413f650 2024-03-29T10:46:30.753 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.757+0000 7f8e0bfff640 1 -- 172.21.15.57:0/1905035311 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e241422a0 con 0x7f8e2413f650 2024-03-29T10:46:30.753 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.757+0000 7f8e0bfff640 1 -- 172.21.15.57:0/1905035311 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f8e18001960 con 0x7f8e2413f650 2024-03-29T10:46:30.753 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.757+0000 7f8e0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:30.753 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.757+0000 7f8e0bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:30.753 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.757+0000 7f8e0bfff640 10 monclient(hunting): dump: 2024-03-29T10:46:30.753 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:30.753 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:30.753 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:30.753 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:30.754 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:30.754 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:30.754 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:30.754 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:30.754 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:30.754 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:30.754 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.757+0000 7f8e0bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:30.754 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.757+0000 7f8e0bfff640 1 -- 172.21.15.57:0/1905035311 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484449957 0 0) 0x7f8e24142e70 con 0x7f8e241457f0 2024-03-29T10:46:30.754 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.757+0000 7f8e0bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:30.754 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.757+0000 7f8e0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:30.754 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.757+0000 7f8e0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:30.754 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.757+0000 7f8e0bfff640 10 monclient(hunting): my global_id is 4462 2024-03-29T10:46:30.754 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.757+0000 7f8e0bfff640 1 -- 172.21.15.57:0/1905035311 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e10004eb0 con 0x7f8e241457f0 2024-03-29T10:46:30.754 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.757+0000 7f8e0bfff640 1 -- 172.21.15.57:0/1905035311 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2057313392 0 0) 0x7f8e241422a0 con 0x7f8e2413f650 2024-03-29T10:46:30.754 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.757+0000 7f8e0bfff640 1 -- 172.21.15.57:0/1905035311 >> v1:172.21.15.179:6789/0 conn(0x7f8e241457f0 legacy=0x7f8e241e2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:30.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.757+0000 7f8e0bfff640 1 -- 172.21.15.57:0/1905035311 >> v1:172.21.15.57:6789/0 conn(0x7f8e240a9d70 legacy=0x7f8e24142760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:46:30.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.758+0000 7f8e0bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:30.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.758+0000 7f8e0bfff640 1 monclient: found mon.c 2024-03-29T10:46:30.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.758+0000 7f8e0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:30.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.758+0000 7f8e0bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:46:30.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.758+0000 7f8e0bfff640 1 -- 172.21.15.57:0/1905035311 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e24132ac0 con 0x7f8e2413f650 2024-03-29T10:46:30.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.758+0000 7f8e0bfff640 10 monclient: _finish_auth 0 2024-03-29T10:46:30.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.758+0000 7f8e0bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:30.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.758+0000 7f8e0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:30.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.758+0000 7f8e29a4e640 5 monclient: authenticate success, global_id 4496 2024-03-29T10:46:30.755 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.758+0000 7f8e29a4e640 10 monclient: _renew_subs 2024-03-29T10:46:30.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.758+0000 7f8e29a4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:46:30.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.758+0000 7f8e29a4e640 1 -- 172.21.15.57:0/1905035311 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e24132da0 con 0x7f8e2413f650 2024-03-29T10:46:30.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.758+0000 7f8e0bfff640 1 -- 172.21.15.57:0/1905035311 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e18001c40 con 0x7f8e2413f650 2024-03-29T10:46:30.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.758+0000 7f8e0bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:30.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.758+0000 7f8e20ff9640 4 set_mon_vals no callback set 2024-03-29T10:46:30.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.758+0000 7f8e29a4e640 10 client.4496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:30.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.758+0000 7f8e29a4e640 10 monclient: _renew_subs 2024-03-29T10:46:30.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.758+0000 7f8e29a4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:46:30.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.758+0000 7f8e29a4e640 1 -- 172.21.15.57:0/1905035311 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e24131ef0 con 0x7f8e2413f650 2024-03-29T10:46:30.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.759+0000 7f8e29a4e640 1 librados: init done 2024-03-29T10:46:30.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.759+0000 7f8e0bfff640 1 -- 172.21.15.57:0/1905035311 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f8e18003a40 con 0x7f8e2413f650 2024-03-29T10:46:30.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.759+0000 7f8e0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:30.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.759+0000 7f8e0bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:30.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.759+0000 7f8e0bfff640 10 monclient: dump: 2024-03-29T10:46:30.757 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:30.757 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:30.757 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:30.757 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:30.757 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:30.757 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:30.757 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:30.757 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:30.757 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:30.757 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:30.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.761+0000 7f8e0bfff640 1 -- 172.21.15.57:0/1905035311 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f8e18003c20 con 0x7f8e2413f650 2024-03-29T10:46:30.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.762+0000 7f8e29a4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:30.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.762+0000 7f8e29a4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:30.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.762+0000 7f8e29a4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:46:30.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.762+0000 7f8e29a4e640 1 -- 172.21.15.57:0/1905035311 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e24142480 con 0x7f8e2413f650 2024-03-29T10:46:30.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.762+0000 7f8e22ffd640 10 monclient: get_auth_request con 0x7f8e1006dd10 auth_method 0 2024-03-29T10:46:30.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.763+0000 7f8e0bfff640 1 -- 172.21.15.57:0/1905035311 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f8e18084b60 con 0x7f8e2413f650 2024-03-29T10:46:30.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.763+0000 7f8e0bfff640 10 client.4496.objecter ms_dispatch 0x7f8e24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:30.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.763+0000 7f8e0bfff640 3 client.4496.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:30.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.763+0000 7f8e0bfff640 3 client.4496.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:30.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.763+0000 7f8e0bfff640 20 client.4496.objecter dump_active .. 0 homeless 2024-03-29T10:46:30.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.763+0000 7f8e0bfff640 10 client.4496.objecter ms_handle_connect 0x7f8e1006dd10 2024-03-29T10:46:30.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.770+0000 7f8e0bfff640 1 -- 172.21.15.57:0/1905035311 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e24142480 con 0x7f8e2413f650 2024-03-29T10:46:30.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.770+0000 7f8e0bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:30.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:30.770+0000 7f8e0bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:31.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:31.082+0000 7f8e29a4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:31.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:31.082+0000 7f8e29a4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:31.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:31.082+0000 7f8e29a4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:46:31.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:31.082+0000 7f8e29a4e640 1 -- 172.21.15.57:0/1905035311 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e24142480 con 0x7f8e2413f650 2024-03-29T10:46:31.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:31.083+0000 7f8e0bfff640 1 -- 172.21.15.57:0/1905035311 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f8e24142480 con 0x7f8e2413f650 2024-03-29T10:46:31.066 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:31.083+0000 7f8e0bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:31.066 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:31.083+0000 7f8e0bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:31.066 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:46:31.066 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:31.069 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:31.086+0000 7f8e29a4e640 10 librados: watch_flush enter 2024-03-29T10:46:31.069 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:31.087+0000 7f8e29a4e640 10 librados: watch_flush exit 2024-03-29T10:46:31.069 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:31.087+0000 7f8e29a4e640 20 client.4496.objecter shutdown clearing up homeless session... 2024-03-29T10:46:31.070 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:31.087+0000 7f8e29a4e640 10 client.4496.objecter successfully canceled tick 2024-03-29T10:46:31.070 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:31.087+0000 7f8e29a4e640 1 -- 172.21.15.57:0/1905035311 >> v1:172.21.15.179:6816/2698295936 conn(0x7f8e1006dd10 legacy=0x7f8e100701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:31.070 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:31.087+0000 7f8e29a4e640 10 monclient: shutdown 2024-03-29T10:46:31.070 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:31.087+0000 7f8e29a4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:31.070 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:31.087+0000 7f8e29a4e640 1 -- 172.21.15.57:0/1905035311 >> v1:172.21.15.57:6790/0 conn(0x7f8e2413f650 legacy=0x7f8e241dedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:31.070 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:31.087+0000 7f8e29a4e640 1 -- 172.21.15.57:0/1905035311 shutdown_connections 2024-03-29T10:46:31.070 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:31.087+0000 7f8e29a4e640 1 -- 172.21.15.57:0/1905035311 >> 172.21.15.57:0/1905035311 conn(0x7f8e24066a00 msgr2=0x7f8e240b4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:31.071 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:31.088+0000 7f8e29a4e640 1 -- 172.21.15.57:0/1905035311 shutdown_connections 2024-03-29T10:46:31.071 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:31.088+0000 7f8e29a4e640 1 -- 172.21.15.57:0/1905035311 wait complete. 2024-03-29T10:46:31.071 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:31.088+0000 7f8e29a4e640 1 librados: shutdown 2024-03-29T10:46:31.084 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:37.891 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:38.039 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.055+0000 7f347a0d0640 10 monclient: get_monmap_and_config 2024-03-29T10:46:38.039 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.055+0000 7f347a0d0640 10 monclient: build_initial_monmap 2024-03-29T10:46:38.039 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.055+0000 7f347a0d0640 10 monclient: monmap: 2024-03-29T10:46:38.039 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:46:38.039 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:38.039 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:46:38.057306+0000 2024-03-29T10:46:38.039 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:46:38.057306+0000 2024-03-29T10:46:38.040 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:38.040 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:38.040 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:46:38.040 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:46:38.040 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:46:38.040 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:38.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.056+0000 7f347a0d0640 1 Processor -- start 2024-03-29T10:46:38.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.056+0000 7f347a0d0640 1 -- start start 2024-03-29T10:46:38.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.056+0000 7f347a0d0640 10 monclient: init 2024-03-29T10:46:38.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.057+0000 7f347a0d0640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:38.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.057+0000 7f347a0d0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:38.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.058+0000 7f347a0d0640 10 monclient(hunting): picked mon.noname-a con 0x7f347414a400 addr v1:172.21.15.57:6789/0 2024-03-29T10:46:38.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.058+0000 7f347a0d0640 10 monclient(hunting): picked mon.noname-c con 0x7f34740aa500 addr v1:172.21.15.179:6789/0 2024-03-29T10:46:38.041 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.058+0000 7f347a0d0640 10 monclient(hunting): picked mon.noname-b con 0x7f3474149800 addr v1:172.21.15.57:6790/0 2024-03-29T10:46:38.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.059+0000 7f347a0d0640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34740aca50 con 0x7f347414a400 2024-03-29T10:46:38.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.059+0000 7f347a0d0640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3474149cb0 con 0x7f34740aa500 2024-03-29T10:46:38.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.059+0000 7f347a0d0640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34740acc80 con 0x7f3474149800 2024-03-29T10:46:38.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.059+0000 7f347a0d0640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:38.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.059+0000 7f347a0d0640 10 monclient(hunting): authenticate will time out at 1350.236379s 2024-03-29T10:46:38.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.059+0000 7f34788cd640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f34740aa500 0x7f34740ad160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:40620/0 (socket says 172.21.15.57:40620) 2024-03-29T10:46:38.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.059+0000 7f34788cd640 1 -- 172.21.15.57:0/1549316494 learned_addr learned my addr 172.21.15.57:0/1549316494 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:46:38.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.060+0000 7f346bfff640 1 -- 172.21.15.57:0/1549316494 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781797443 0 0) 0x7f3474149cb0 con 0x7f34740aa500 2024-03-29T10:46:38.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.060+0000 7f346bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:46:38.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.060+0000 7f346bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:38.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.060+0000 7f346bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:38.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.060+0000 7f346bfff640 10 monclient(hunting): my global_id is 4465 2024-03-29T10:46:38.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.060+0000 7f346bfff640 1 -- 172.21.15.57:0/1549316494 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3460003520 con 0x7f34740aa500 2024-03-29T10:46:38.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.060+0000 7f346bfff640 1 -- 172.21.15.57:0/1549316494 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1502468258 0 0) 0x7f3460003520 con 0x7f34740aa500 2024-03-29T10:46:38.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.060+0000 7f346bfff640 1 -- 172.21.15.57:0/1549316494 >> v1:172.21.15.57:6790/0 conn(0x7f3474149800 legacy=0x7f3474151ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:38.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.062+0000 7f346bfff640 1 -- 172.21.15.57:0/1549316494 >> v1:172.21.15.57:6789/0 conn(0x7f347414a400 legacy=0x7f34740aac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:38.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.062+0000 7f346bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:46:38.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.062+0000 7f346bfff640 1 monclient: found mon.noname-c 2024-03-29T10:46:38.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.062+0000 7f346bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:38.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.062+0000 7f346bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:46:38.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.062+0000 7f346bfff640 1 -- 172.21.15.57:0/1549316494 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34740ac350 con 0x7f34740aa500 2024-03-29T10:46:38.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.062+0000 7f346bfff640 10 monclient: _finish_auth 0 2024-03-29T10:46:38.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.062+0000 7f346bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:46:38.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.062+0000 7f346bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:38.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.062+0000 7f346bfff640 1 -- 172.21.15.57:0/1549316494 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f3470001690 con 0x7f34740aa500 2024-03-29T10:46:38.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.062+0000 7f346bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:38.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.062+0000 7f346bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:38.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.062+0000 7f346bfff640 10 monclient: dump: 2024-03-29T10:46:38.047 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:38.047 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:38.047 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:38.047 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:38.047 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:38.047 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:38.047 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:38.047 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:38.047 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:38.047 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:38.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.062+0000 7f347a0d0640 5 monclient: authenticate success, global_id 4465 2024-03-29T10:46:38.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.062+0000 7f347a0d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:38.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.065+0000 7f346bfff640 1 -- 172.21.15.57:0/1549316494 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3470001860 con 0x7f34740aa500 2024-03-29T10:46:38.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.065+0000 7f346bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:38.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.065+0000 7f346bfff640 1 -- 172.21.15.57:0/1549316494 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f3470001c70 con 0x7f34740aa500 2024-03-29T10:46:38.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.065+0000 7f346bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:38.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.065+0000 7f346bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:38.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.065+0000 7f346bfff640 10 monclient: dump: 2024-03-29T10:46:38.049 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:38.049 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:38.049 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:38.049 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:38.049 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:38.049 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:38.049 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:38.050 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:38.050 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:38.050 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:38.050 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.065+0000 7f347a0d0640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:38.050 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.066+0000 7f347a0d0640 4 set_mon_vals no callback set 2024-03-29T10:46:38.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.067+0000 7f347a0d0640 10 monclient: shutdown 2024-03-29T10:46:38.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.067+0000 7f347a0d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:38.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.067+0000 7f347a0d0640 1 -- 172.21.15.57:0/1549316494 >> v1:172.21.15.179:6789/0 conn(0x7f34740aa500 legacy=0x7f34740ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:38.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.067+0000 7f347a0d0640 1 -- 172.21.15.57:0/1549316494 shutdown_connections 2024-03-29T10:46:38.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.067+0000 7f347a0d0640 1 -- 172.21.15.57:0/1549316494 >> 172.21.15.57:0/1549316494 conn(0x7f34740b3a30 msgr2=0x7f34740b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:38.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.067+0000 7f347a0d0640 1 -- 172.21.15.57:0/1549316494 shutdown_connections 2024-03-29T10:46:38.051 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.067+0000 7f347a0d0640 1 -- 172.21.15.57:0/1549316494 wait complete. 2024-03-29T10:46:38.054 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.068+0000 7f347a0d0640 10 monclient: build_initial_monmap 2024-03-29T10:46:38.054 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.068+0000 7f347a0d0640 10 monclient: monmap: 2024-03-29T10:46:38.054 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:46:38.054 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:38.054 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:46:38.054 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:46:38.054 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:38.054 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:38.054 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:46:38.054 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:46:38.054 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:46:38.054 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:38.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.069+0000 7f347a0d0640 1 librados: starting msgr at 2024-03-29T10:46:38.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.069+0000 7f347a0d0640 1 librados: starting objecter 2024-03-29T10:46:38.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.069+0000 7f347a0d0640 1 Processor -- start 2024-03-29T10:46:38.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.069+0000 7f347a0d0640 1 -- start start 2024-03-29T10:46:38.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.069+0000 7f347a0d0640 1 librados: setting wanted keys 2024-03-29T10:46:38.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.069+0000 7f347a0d0640 1 librados: calling monclient init 2024-03-29T10:46:38.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.069+0000 7f347a0d0640 10 monclient: init 2024-03-29T10:46:38.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.070+0000 7f347a0d0640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:38.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.070+0000 7f347a0d0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:46:38.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.070+0000 7f347a0d0640 10 monclient(hunting): picked mon.noname-b con 0x7f34740aa500 addr v1:172.21.15.179:6789/0 2024-03-29T10:46:38.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.070+0000 7f347a0d0640 10 monclient(hunting): picked mon.noname-a con 0x7f3474149800 addr v1:172.21.15.57:6789/0 2024-03-29T10:46:38.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.070+0000 7f347a0d0640 10 monclient(hunting): picked mon.noname-c con 0x7f347414a400 addr v1:172.21.15.57:6790/0 2024-03-29T10:46:38.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.070+0000 7f347a0d0640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34741485a0 con 0x7f3474149800 2024-03-29T10:46:38.056 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.071+0000 7f34790ce640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f34740aa500 0x7f3474146d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:41808/0 (socket says 172.21.15.57:41808) 2024-03-29T10:46:38.056 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.071+0000 7f34790ce640 1 -- 172.21.15.57:0/2308321675 learned_addr learned my addr 172.21.15.57:0/2308321675 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:46:38.062 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.071+0000 7f347a0d0640 1 -- 172.21.15.57:0/2308321675 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347413eed0 con 0x7f34740aa500 2024-03-29T10:46:38.062 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.071+0000 7f347a0d0640 1 -- 172.21.15.57:0/2308321675 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34741452f0 con 0x7f347414a400 2024-03-29T10:46:38.062 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.071+0000 7f347a0d0640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:38.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.071+0000 7f347a0d0640 10 monclient(hunting): authenticate will time out at 1350.248903s 2024-03-29T10:46:38.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.074+0000 7f3469ffb640 10 client.?.objecter ms_handle_connect 0x7f347414a400 2024-03-29T10:46:38.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.074+0000 7f3469ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:38.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.074+0000 7f3469ffb640 1 -- 172.21.15.57:0/2308321675 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 320976876 0 0) 0x7f34741452f0 con 0x7f347414a400 2024-03-29T10:46:38.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.075+0000 7f3469ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:38.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.075+0000 7f3469ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:38.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.075+0000 7f3469ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:38.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.075+0000 7f3469ffb640 10 monclient(hunting): my global_id is 4499 2024-03-29T10:46:38.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.075+0000 7f3469ffb640 1 -- 172.21.15.57:0/2308321675 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3458004eb0 con 0x7f347414a400 2024-03-29T10:46:38.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.075+0000 7f3469ffb640 10 client.?.objecter ms_handle_connect 0x7f34740aa500 2024-03-29T10:46:38.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.075+0000 7f3469ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:38.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.075+0000 7f3469ffb640 1 -- 172.21.15.57:0/2308321675 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 147060714 0 0) 0x7f347413eed0 con 0x7f34740aa500 2024-03-29T10:46:38.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.075+0000 7f3469ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:38.063 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.075+0000 7f3469ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:38.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.075+0000 7f3469ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:38.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.075+0000 7f3469ffb640 10 monclient(hunting): my global_id is 4468 2024-03-29T10:46:38.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.075+0000 7f3469ffb640 1 -- 172.21.15.57:0/2308321675 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34741452f0 con 0x7f34740aa500 2024-03-29T10:46:38.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.076+0000 7f3469ffb640 1 -- 172.21.15.57:0/2308321675 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3336261521 0 0) 0x7f3458004eb0 con 0x7f347414a400 2024-03-29T10:46:38.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.076+0000 7f3469ffb640 1 -- 172.21.15.57:0/2308321675 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f347413eed0 con 0x7f347414a400 2024-03-29T10:46:38.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.076+0000 7f3469ffb640 1 -- 172.21.15.57:0/2308321675 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f346c001b70 con 0x7f347414a400 2024-03-29T10:46:38.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.076+0000 7f3469ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:38.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.076+0000 7f3469ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:38.064 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.076+0000 7f3469ffb640 10 monclient(hunting): dump: 2024-03-29T10:46:38.064 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:38.064 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:38.064 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:38.064 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:38.064 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:38.064 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:38.065 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:38.065 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:38.065 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:38.065 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:38.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.076+0000 7f3469ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:38.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.076+0000 7f3469ffb640 10 client.?.objecter ms_handle_connect 0x7f3474149800 2024-03-29T10:46:38.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.076+0000 7f3469ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:38.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.076+0000 7f3469ffb640 1 -- 172.21.15.57:0/2308321675 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 12284633 0 0) 0x7f34741452f0 con 0x7f34740aa500 2024-03-29T10:46:38.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.076+0000 7f3469ffb640 1 -- 172.21.15.57:0/2308321675 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3458004eb0 con 0x7f34740aa500 2024-03-29T10:46:38.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.076+0000 7f3469ffb640 1 -- 172.21.15.57:0/2308321675 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693439966 0 0) 0x7f34741485a0 con 0x7f3474149800 2024-03-29T10:46:38.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.076+0000 7f3469ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:46:38.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.077+0000 7f3469ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:38.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.077+0000 7f3469ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:38.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.077+0000 7f3469ffb640 10 monclient(hunting): my global_id is 4458 2024-03-29T10:46:38.065 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.077+0000 7f3469ffb640 1 -- 172.21.15.57:0/2308321675 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34741452f0 con 0x7f3474149800 2024-03-29T10:46:38.066 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.077+0000 7f3469ffb640 1 -- 172.21.15.57:0/2308321675 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2233783855 0 0) 0x7f347413eed0 con 0x7f347414a400 2024-03-29T10:46:38.066 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.077+0000 7f3469ffb640 1 -- 172.21.15.57:0/2308321675 >> v1:172.21.15.179:6789/0 conn(0x7f34740aa500 legacy=0x7f3474146d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:38.066 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.077+0000 7f3469ffb640 1 -- 172.21.15.57:0/2308321675 >> v1:172.21.15.57:6789/0 conn(0x7f3474149800 legacy=0x7f3474144100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:38.066 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.077+0000 7f3469ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:46:38.066 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.077+0000 7f3469ffb640 1 monclient: found mon.c 2024-03-29T10:46:38.066 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.077+0000 7f3469ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:38.066 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.077+0000 7f3469ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:46:38.066 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.077+0000 7f3469ffb640 1 -- 172.21.15.57:0/2308321675 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3474144f20 con 0x7f347414a400 2024-03-29T10:46:38.066 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.077+0000 7f3469ffb640 10 monclient: _finish_auth 0 2024-03-29T10:46:38.066 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.077+0000 7f3469ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:46:38.066 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.077+0000 7f3469ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:38.066 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.078+0000 7f347a0d0640 5 monclient: authenticate success, global_id 4499 2024-03-29T10:46:38.066 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.078+0000 7f347a0d0640 10 monclient: _renew_subs 2024-03-29T10:46:38.066 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.078+0000 7f347a0d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:46:38.067 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.078+0000 7f347a0d0640 1 -- 172.21.15.57:0/2308321675 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3474145810 con 0x7f347414a400 2024-03-29T10:46:38.067 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.078+0000 7f347a0d0640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:38.067 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.078+0000 7f347a0d0640 10 monclient: _renew_subs 2024-03-29T10:46:38.067 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.078+0000 7f347a0d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:46:38.067 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.078+0000 7f347a0d0640 1 -- 172.21.15.57:0/2308321675 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3474147660 con 0x7f347414a400 2024-03-29T10:46:38.067 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.078+0000 7f347a0d0640 1 librados: init done 2024-03-29T10:46:38.067 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.079+0000 7f3469ffb640 1 -- 172.21.15.57:0/2308321675 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f346c001cd0 con 0x7f347414a400 2024-03-29T10:46:38.068 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.079+0000 7f347a0d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:38.068 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.079+0000 7f347a0d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:38.068 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.079+0000 7f347a0d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:46:38.068 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.079+0000 7f347a0d0640 1 -- 172.21.15.57:0/2308321675 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f347404f730 con 0x7f347414a400 2024-03-29T10:46:38.068 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.079+0000 7f3469ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:38.068 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.079+0000 7f3469ffb640 1 -- 172.21.15.57:0/2308321675 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f346c003b10 con 0x7f347414a400 2024-03-29T10:46:38.068 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.079+0000 7f3469ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:38.069 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.079+0000 7f3469ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:38.069 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.079+0000 7f3469ffb640 10 monclient: dump: 2024-03-29T10:46:38.069 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:38.069 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:38.069 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:38.069 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:38.069 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:38.069 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:38.069 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:38.069 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:38.069 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:38.069 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:38.069 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.079+0000 7f346b7fe640 4 set_mon_vals no callback set 2024-03-29T10:46:38.069 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.080+0000 7f3469ffb640 1 -- 172.21.15.57:0/2308321675 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f346c019ae0 con 0x7f347414a400 2024-03-29T10:46:38.070 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.081+0000 7f34790ce640 10 monclient: get_auth_request con 0x7f345806df10 auth_method 0 2024-03-29T10:46:38.070 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.081+0000 7f3469ffb640 1 -- 172.21.15.57:0/2308321675 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f346c0839f0 con 0x7f347414a400 2024-03-29T10:46:38.070 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.081+0000 7f3469ffb640 10 client.4499.objecter ms_dispatch 0x7f3474000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:38.070 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.082+0000 7f3469ffb640 3 client.4499.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:38.070 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.082+0000 7f3469ffb640 3 client.4499.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:38.070 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.082+0000 7f3469ffb640 20 client.4499.objecter dump_active .. 0 homeless 2024-03-29T10:46:38.070 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.082+0000 7f3469ffb640 10 client.4499.objecter ms_handle_connect 0x7f345806df10 2024-03-29T10:46:38.071 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.087+0000 7f3469ffb640 1 -- 172.21.15.57:0/2308321675 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f346c04f100 con 0x7f347414a400 2024-03-29T10:46:38.071 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.087+0000 7f3469ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:38.071 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.087+0000 7f3469ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:38.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.428+0000 7f347a0d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:38.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.428+0000 7f347a0d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:38.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.428+0000 7f347a0d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:46:38.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.428+0000 7f347a0d0640 1 -- 172.21.15.57:0/2308321675 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34741450b0 con 0x7f347414a400 2024-03-29T10:46:38.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.430+0000 7f3469ffb640 1 -- 172.21.15.57:0/2308321675 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f346c051200 con 0x7f347414a400 2024-03-29T10:46:38.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.430+0000 7f3469ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:38.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.430+0000 7f3469ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:38.414 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:46:38.414 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:38.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.433+0000 7f347a0d0640 10 librados: watch_flush enter 2024-03-29T10:46:38.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.433+0000 7f347a0d0640 10 librados: watch_flush exit 2024-03-29T10:46:38.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.434+0000 7f347a0d0640 20 client.4499.objecter shutdown clearing up homeless session... 2024-03-29T10:46:38.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.434+0000 7f347a0d0640 10 client.4499.objecter successfully canceled tick 2024-03-29T10:46:38.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.434+0000 7f347a0d0640 1 -- 172.21.15.57:0/2308321675 >> v1:172.21.15.179:6816/2698295936 conn(0x7f345806df10 legacy=0x7f34580703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:38.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.434+0000 7f347a0d0640 10 monclient: shutdown 2024-03-29T10:46:38.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.434+0000 7f347a0d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:38.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.434+0000 7f347a0d0640 1 -- 172.21.15.57:0/2308321675 >> v1:172.21.15.57:6790/0 conn(0x7f347414a400 legacy=0x7f3474144810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:38.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.434+0000 7f347a0d0640 1 -- 172.21.15.57:0/2308321675 shutdown_connections 2024-03-29T10:46:38.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.434+0000 7f347a0d0640 1 -- 172.21.15.57:0/2308321675 >> 172.21.15.57:0/2308321675 conn(0x7f34740b3a30 msgr2=0x7f34740ae430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:38.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.435+0000 7f347a0d0640 1 -- 172.21.15.57:0/2308321675 shutdown_connections 2024-03-29T10:46:38.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.435+0000 7f347a0d0640 1 -- 172.21.15.57:0/2308321675 wait complete. 2024-03-29T10:46:38.419 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:38.435+0000 7f347a0d0640 1 librados: shutdown 2024-03-29T10:46:38.434 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:45.247 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:45.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.418+0000 7f273464a640 10 monclient: get_monmap_and_config 2024-03-29T10:46:45.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.418+0000 7f273464a640 10 monclient: build_initial_monmap 2024-03-29T10:46:45.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.418+0000 7f273464a640 10 monclient: monmap: 2024-03-29T10:46:45.403 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:46:45.403 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:45.403 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:46:45.419791+0000 2024-03-29T10:46:45.403 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:46:45.419791+0000 2024-03-29T10:46:45.403 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:45.403 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:45.403 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:46:45.403 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:46:45.403 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:46:45.403 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:45.404 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.420+0000 7f273464a640 1 Processor -- start 2024-03-29T10:46:45.404 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.420+0000 7f273464a640 1 -- start start 2024-03-29T10:46:45.404 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.420+0000 7f273464a640 10 monclient: init 2024-03-29T10:46:45.405 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.421+0000 7f273464a640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:45.406 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.421+0000 7f273464a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:46:45.406 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.421+0000 7f273464a640 10 monclient(hunting): picked mon.noname-c con 0x7f272c0ae250 addr v1:172.21.15.179:6789/0 2024-03-29T10:46:45.406 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.421+0000 7f273464a640 10 monclient(hunting): picked mon.noname-a con 0x7f272c13f970 addr v1:172.21.15.57:6789/0 2024-03-29T10:46:45.406 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.422+0000 7f273464a640 10 monclient(hunting): picked mon.noname-b con 0x7f272c142800 addr v1:172.21.15.57:6790/0 2024-03-29T10:46:45.406 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.422+0000 7f273464a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f272c0afe00 con 0x7f272c13f970 2024-03-29T10:46:45.406 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.422+0000 7f273464a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f272c142c10 con 0x7f272c0ae250 2024-03-29T10:46:45.406 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.422+0000 7f273464a640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f272c0afb20 con 0x7f272c142800 2024-03-29T10:46:45.406 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.422+0000 7f273464a640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:45.406 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.422+0000 7f273464a640 10 monclient(hunting): authenticate will time out at 1357.599070s 2024-03-29T10:46:45.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.426+0000 7f2732be7640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f272c142800 0x7f272c14da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:52301/0 (socket says 172.21.15.57:52301) 2024-03-29T10:46:45.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.426+0000 7f2732be7640 1 -- 172.21.15.57:0/4061733602 learned_addr learned my addr 172.21.15.57:0/4061733602 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:46:45.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.429+0000 7f27313e4640 1 -- 172.21.15.57:0/4061733602 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615145966 0 0) 0x7f272c0afb20 con 0x7f272c142800 2024-03-29T10:46:45.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.429+0000 7f27313e4640 10 monclient(hunting): _negotiate 2024-03-29T10:46:45.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.429+0000 7f27313e4640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:45.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.429+0000 7f27313e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:45.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.429+0000 7f27313e4640 10 monclient(hunting): my global_id is 4502 2024-03-29T10:46:45.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.429+0000 7f27313e4640 1 -- 172.21.15.57:0/4061733602 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2718003520 con 0x7f272c142800 2024-03-29T10:46:45.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.429+0000 7f27313e4640 1 -- 172.21.15.57:0/4061733602 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586047235 0 0) 0x7f272c142c10 con 0x7f272c0ae250 2024-03-29T10:46:45.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.429+0000 7f27313e4640 10 monclient(hunting): _negotiate 2024-03-29T10:46:45.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.429+0000 7f27313e4640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:45.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.429+0000 7f27313e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:45.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.429+0000 7f27313e4640 10 monclient(hunting): my global_id is 4471 2024-03-29T10:46:45.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.429+0000 7f27313e4640 1 -- 172.21.15.57:0/4061733602 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f272c0afb20 con 0x7f272c0ae250 2024-03-29T10:46:45.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.430+0000 7f27313e4640 1 -- 172.21.15.57:0/4061733602 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727532460 0 0) 0x7f272c0afe00 con 0x7f272c13f970 2024-03-29T10:46:45.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.430+0000 7f27313e4640 10 monclient(hunting): _negotiate 2024-03-29T10:46:45.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.430+0000 7f27313e4640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:45.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.430+0000 7f27313e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:45.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.430+0000 7f27313e4640 10 monclient(hunting): my global_id is 4461 2024-03-29T10:46:45.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.430+0000 7f27313e4640 1 -- 172.21.15.57:0/4061733602 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f272c142c10 con 0x7f272c13f970 2024-03-29T10:46:45.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.430+0000 7f27313e4640 1 -- 172.21.15.57:0/4061733602 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3253600764 0 0) 0x7f272c142c10 con 0x7f272c13f970 2024-03-29T10:46:45.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.430+0000 7f27313e4640 1 -- 172.21.15.57:0/4061733602 >> v1:172.21.15.57:6790/0 conn(0x7f272c142800 legacy=0x7f272c14da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:45.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.430+0000 7f27313e4640 1 -- 172.21.15.57:0/4061733602 >> v1:172.21.15.179:6789/0 conn(0x7f272c0ae250 legacy=0x7f272c13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:45.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.430+0000 7f27313e4640 10 monclient: _finish_hunting 0 2024-03-29T10:46:45.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.430+0000 7f27313e4640 1 monclient: found mon.noname-a 2024-03-29T10:46:45.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.430+0000 7f27313e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:45.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.430+0000 7f27313e4640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:46:45.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.430+0000 7f27313e4640 1 -- 172.21.15.57:0/4061733602 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272c0aab80 con 0x7f272c13f970 2024-03-29T10:46:45.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.430+0000 7f27313e4640 10 monclient: _finish_auth 0 2024-03-29T10:46:45.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.430+0000 7f27313e4640 10 monclient: _check_auth_tickets 2024-03-29T10:46:45.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.430+0000 7f27313e4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:45.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.430+0000 7f273464a640 5 monclient: authenticate success, global_id 4461 2024-03-29T10:46:45.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.430+0000 7f273464a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:45.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.430+0000 7f27313e4640 1 -- 172.21.15.57:0/4061733602 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f2728001570 con 0x7f272c13f970 2024-03-29T10:46:45.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.430+0000 7f27313e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:45.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.431+0000 7f27313e4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:46:45.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.431+0000 7f27313e4640 10 monclient: dump: 2024-03-29T10:46:45.416 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:45.416 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:45.416 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:45.416 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:45.416 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:45.416 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:45.416 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:45.416 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:45.416 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:45.416 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:45.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.431+0000 7f273464a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:46:45.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.431+0000 7f27313e4640 1 -- 172.21.15.57:0/4061733602 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2728001800 con 0x7f272c13f970 2024-03-29T10:46:45.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.431+0000 7f27313e4640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:45.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.431+0000 7f273464a640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:45.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.431+0000 7f273464a640 4 set_mon_vals no callback set 2024-03-29T10:46:45.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.432+0000 7f273464a640 10 monclient: shutdown 2024-03-29T10:46:45.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.432+0000 7f273464a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:45.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.432+0000 7f273464a640 1 -- 172.21.15.57:0/4061733602 >> v1:172.21.15.57:6789/0 conn(0x7f272c13f970 legacy=0x7f272c1420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:45.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.432+0000 7f273464a640 1 -- 172.21.15.57:0/4061733602 shutdown_connections 2024-03-29T10:46:45.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.432+0000 7f273464a640 1 -- 172.21.15.57:0/4061733602 >> 172.21.15.57:0/4061733602 conn(0x7f272c0b35e0 msgr2=0x7f272c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:45.419 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.436+0000 7f273464a640 1 -- 172.21.15.57:0/4061733602 shutdown_connections 2024-03-29T10:46:45.421 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.437+0000 7f273464a640 1 -- 172.21.15.57:0/4061733602 wait complete. 2024-03-29T10:46:45.422 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.438+0000 7f273464a640 10 monclient: build_initial_monmap 2024-03-29T10:46:45.423 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.438+0000 7f273464a640 10 monclient: monmap: 2024-03-29T10:46:45.423 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:46:45.423 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:45.423 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:46:45.423 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:46:45.423 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:45.423 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:45.423 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:46:45.423 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:46:45.423 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:46:45.423 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:45.424 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.438+0000 7f273464a640 1 librados: starting msgr at 2024-03-29T10:46:45.424 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.438+0000 7f273464a640 1 librados: starting objecter 2024-03-29T10:46:45.424 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.438+0000 7f273464a640 1 Processor -- start 2024-03-29T10:46:45.425 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.439+0000 7f273464a640 1 -- start start 2024-03-29T10:46:45.425 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.439+0000 7f273464a640 1 librados: setting wanted keys 2024-03-29T10:46:45.425 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.439+0000 7f273464a640 1 librados: calling monclient init 2024-03-29T10:46:45.425 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.439+0000 7f273464a640 10 monclient: init 2024-03-29T10:46:45.425 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.440+0000 7f273464a640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:45.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.440+0000 7f273464a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:46:45.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.440+0000 7f273464a640 10 monclient(hunting): picked mon.noname-c con 0x7f272c142800 addr v1:172.21.15.57:6790/0 2024-03-29T10:46:45.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.440+0000 7f273464a640 10 monclient(hunting): picked mon.noname-a con 0x7f272c141310 addr v1:172.21.15.57:6789/0 2024-03-29T10:46:45.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.440+0000 7f273464a640 10 monclient(hunting): picked mon.noname-b con 0x7f272c145950 addr v1:172.21.15.179:6789/0 2024-03-29T10:46:45.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.440+0000 7f273464a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f272c145540 con 0x7f272c141310 2024-03-29T10:46:45.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.440+0000 7f273464a640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f272c145da0 con 0x7f272c145950 2024-03-29T10:46:45.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.440+0000 7f273464a640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f272c06e8d0 con 0x7f272c142800 2024-03-29T10:46:45.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.440+0000 7f273464a640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:45.426 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.440+0000 7f273464a640 10 monclient(hunting): authenticate will time out at 1357.617636s 2024-03-29T10:46:45.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.444+0000 7f2732be7640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f272c145950 0x7f272c1e5f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:44917/0 (socket says 172.21.15.57:44917) 2024-03-29T10:46:45.427 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.444+0000 7f2732be7640 1 -- 172.21.15.57:0/1450674660 learned_addr learned my addr 172.21.15.57:0/1450674660 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:46:45.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.447+0000 7f2722ffd640 10 client.?.objecter ms_handle_connect 0x7f272c145950 2024-03-29T10:46:45.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.447+0000 7f2722ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:45.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.447+0000 7f2722ffd640 1 -- 172.21.15.57:0/1450674660 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690399217 0 0) 0x7f272c145da0 con 0x7f272c145950 2024-03-29T10:46:45.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.447+0000 7f2722ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:45.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.447+0000 7f2722ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:45.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.447+0000 7f2722ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:45.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.447+0000 7f2722ffd640 10 monclient(hunting): my global_id is 4474 2024-03-29T10:46:45.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.448+0000 7f2722ffd640 1 -- 172.21.15.57:0/1450674660 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2710004eb0 con 0x7f272c145950 2024-03-29T10:46:45.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.448+0000 7f2722ffd640 10 client.?.objecter ms_handle_connect 0x7f272c141310 2024-03-29T10:46:45.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.448+0000 7f2722ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:45.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.448+0000 7f2722ffd640 1 -- 172.21.15.57:0/1450674660 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3843635010 0 0) 0x7f2710004eb0 con 0x7f272c145950 2024-03-29T10:46:45.434 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.448+0000 7f2722ffd640 1 -- 172.21.15.57:0/1450674660 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f272c145da0 con 0x7f272c145950 2024-03-29T10:46:45.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.449+0000 7f2722ffd640 1 -- 172.21.15.57:0/1450674660 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f2724001b70 con 0x7f272c145950 2024-03-29T10:46:45.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.449+0000 7f2722ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:45.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.449+0000 7f2722ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:45.435 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.449+0000 7f2722ffd640 10 monclient(hunting): dump: 2024-03-29T10:46:45.435 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:45.435 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:45.435 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:45.435 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:45.435 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:45.435 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:45.435 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:45.435 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:45.435 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:45.435 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:45.436 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.449+0000 7f2722ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:45.436 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.449+0000 7f2722ffd640 1 -- 172.21.15.57:0/1450674660 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1057751008 0 0) 0x7f272c145da0 con 0x7f272c145950 2024-03-29T10:46:45.436 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.449+0000 7f2722ffd640 1 -- 172.21.15.57:0/1450674660 >> v1:172.21.15.57:6790/0 conn(0x7f272c142800 legacy=0x7f272c1deaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:45.436 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.449+0000 7f2722ffd640 1 -- 172.21.15.57:0/1450674660 >> v1:172.21.15.57:6789/0 conn(0x7f272c141310 legacy=0x7f272c1df200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:45.436 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.449+0000 7f2722ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:46:45.436 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.449+0000 7f2722ffd640 1 monclient: found mon.b 2024-03-29T10:46:45.436 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.449+0000 7f2722ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:45.436 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.449+0000 7f2722ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:46:45.436 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.449+0000 7f2722ffd640 1 -- 172.21.15.57:0/1450674660 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272c06ebb0 con 0x7f272c145950 2024-03-29T10:46:45.436 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.450+0000 7f2722ffd640 10 monclient: _finish_auth 0 2024-03-29T10:46:45.436 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.450+0000 7f2722ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:46:45.436 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.450+0000 7f2722ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:45.436 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.450+0000 7f273464a640 5 monclient: authenticate success, global_id 4474 2024-03-29T10:46:45.436 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.450+0000 7f273464a640 10 monclient: _renew_subs 2024-03-29T10:46:45.436 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.450+0000 7f273464a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:46:45.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.450+0000 7f273464a640 1 -- 172.21.15.57:0/1450674660 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f272c1e7650 con 0x7f272c145950 2024-03-29T10:46:45.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.450+0000 7f273464a640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:45.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.450+0000 7f273464a640 10 monclient: _renew_subs 2024-03-29T10:46:45.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.450+0000 7f273464a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:46:45.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.450+0000 7f273464a640 1 -- 172.21.15.57:0/1450674660 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f272c1e7b80 con 0x7f272c145950 2024-03-29T10:46:45.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.450+0000 7f273464a640 1 librados: init done 2024-03-29T10:46:45.438 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.452+0000 7f2722ffd640 1 -- 172.21.15.57:0/1450674660 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27240020a0 con 0x7f272c145950 2024-03-29T10:46:45.438 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.452+0000 7f2722ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:45.438 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.452+0000 7f2722ffd640 1 -- 172.21.15.57:0/1450674660 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f2724004d40 con 0x7f272c145950 2024-03-29T10:46:45.438 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.452+0000 7f2722ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:45.438 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.452+0000 7f2722ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:46:45.439 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.452+0000 7f2722ffd640 10 monclient: dump: 2024-03-29T10:46:45.439 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:45.439 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:45.439 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:45.439 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:45.439 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:45.439 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:45.439 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:45.439 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:45.439 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:45.439 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:45.439 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.452+0000 7f2723fff640 4 set_mon_vals no callback set 2024-03-29T10:46:45.439 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.452+0000 7f2720ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:45.439 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.453+0000 7f2720ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:45.440 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.453+0000 7f2720ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:46:45.440 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.453+0000 7f2720ff9640 1 -- 172.21.15.57:0/1450674660 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2714005080 con 0x7f272c145950 2024-03-29T10:46:45.440 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.455+0000 7f2722ffd640 1 -- 172.21.15.57:0/1450674660 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f2724004fa0 con 0x7f272c145950 2024-03-29T10:46:45.440 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.456+0000 7f2722ffd640 1 -- 172.21.15.57:0/1450674660 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f2724084db0 con 0x7f272c145950 2024-03-29T10:46:45.440 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.456+0000 7f2722ffd640 10 client.4474.objecter ms_dispatch 0x7f272c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:45.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.456+0000 7f2722ffd640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:45.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.456+0000 7f2722ffd640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:45.441 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.456+0000 7f2722ffd640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-29T10:46:45.442 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.458+0000 7f27323e6640 10 monclient: get_auth_request con 0x7f271006dc50 auth_method 0 2024-03-29T10:46:45.442 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.459+0000 7f2722ffd640 10 client.4474.objecter ms_handle_connect 0x7f271006dc50 2024-03-29T10:46:45.449 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.465+0000 7f2722ffd640 1 -- 172.21.15.57:0/1450674660 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2724050690 con 0x7f272c145950 2024-03-29T10:46:45.449 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.466+0000 7f2722ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:45.449 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.466+0000 7f2722ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:45.788 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.805+0000 7f2720ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:45.788 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.805+0000 7f2720ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:45.788 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.805+0000 7f2720ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:46:45.789 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.805+0000 7f2720ff9640 1 -- 172.21.15.57:0/1450674660 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2714005330 con 0x7f272c145950 2024-03-29T10:46:45.790 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.807+0000 7f2722ffd640 1 -- 172.21.15.57:0/1450674660 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f27240525c0 con 0x7f272c145950 2024-03-29T10:46:45.790 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.807+0000 7f2722ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:45.790 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.807+0000 7f2722ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:45.792 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:46:45.792 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:45.795 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.812+0000 7f2720ff9640 10 librados: watch_flush enter 2024-03-29T10:46:45.796 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.813+0000 7f2720ff9640 10 librados: watch_flush exit 2024-03-29T10:46:45.796 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.813+0000 7f2720ff9640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-29T10:46:45.796 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.813+0000 7f2720ff9640 10 client.4474.objecter successfully canceled tick 2024-03-29T10:46:45.798 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.815+0000 7f2720ff9640 1 -- 172.21.15.57:0/1450674660 >> v1:172.21.15.179:6816/2698295936 conn(0x7f271006dc50 legacy=0x7f2710070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:45.798 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.815+0000 7f2720ff9640 10 monclient: shutdown 2024-03-29T10:46:45.798 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.815+0000 7f2720ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:45.798 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.815+0000 7f2720ff9640 1 -- 172.21.15.57:0/1450674660 >> v1:172.21.15.179:6789/0 conn(0x7f272c145950 legacy=0x7f272c1e5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:45.799 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.815+0000 7f2720ff9640 1 -- 172.21.15.57:0/1450674660 shutdown_connections 2024-03-29T10:46:45.799 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.815+0000 7f2720ff9640 1 -- 172.21.15.57:0/1450674660 >> 172.21.15.57:0/1450674660 conn(0x7f272c0b35e0 msgr2=0x7f272c0b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:45.800 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.817+0000 7f2720ff9640 1 -- 172.21.15.57:0/1450674660 shutdown_connections 2024-03-29T10:46:45.803 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.820+0000 7f2720ff9640 1 -- 172.21.15.57:0/1450674660 wait complete. 2024-03-29T10:46:45.804 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:45.820+0000 7f2720ff9640 1 librados: shutdown 2024-03-29T10:46:45.823 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:52.631 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:46:52.780 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.796+0000 7f82bb59e640 10 monclient: get_monmap_and_config 2024-03-29T10:46:52.780 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.796+0000 7f82bb59e640 10 monclient: build_initial_monmap 2024-03-29T10:46:52.780 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.796+0000 7f82bb59e640 10 monclient: monmap: 2024-03-29T10:46:52.780 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:46:52.780 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:52.780 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:46:52.798491+0000 2024-03-29T10:46:52.780 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:46:52.798491+0000 2024-03-29T10:46:52.780 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:52.780 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:52.780 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:46:52.781 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:46:52.781 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:46:52.781 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:52.782 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.799+0000 7f82bb59e640 1 Processor -- start 2024-03-29T10:46:52.782 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.799+0000 7f82bb59e640 1 -- start start 2024-03-29T10:46:52.782 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.799+0000 7f82bb59e640 10 monclient: init 2024-03-29T10:46:52.783 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.800+0000 7f82bb59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:52.783 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.800+0000 7f82bb59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:46:52.783 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.800+0000 7f82bb59e640 10 monclient(hunting): picked mon.noname-c con 0x7f82b40389e0 addr v1:172.21.15.179:6789/0 2024-03-29T10:46:52.784 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.800+0000 7f82bb59e640 10 monclient(hunting): picked mon.noname-a con 0x7f82b4038dc0 addr v1:172.21.15.57:6789/0 2024-03-29T10:46:52.787 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.800+0000 7f82bb59e640 10 monclient(hunting): picked mon.noname-b con 0x7f82b4030660 addr v1:172.21.15.57:6790/0 2024-03-29T10:46:52.788 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.804+0000 7f82b9d9b640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f82b4038dc0 0x7f82b40cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:42231/0 (socket says 172.21.15.57:42231) 2024-03-29T10:46:52.788 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.804+0000 7f82bb59e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b40c54c0 con 0x7f82b4038dc0 2024-03-29T10:46:52.788 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.804+0000 7f82bb59e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b4036430 con 0x7f82b40389e0 2024-03-29T10:46:52.788 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.804+0000 7f82bb59e640 1 -- 172.21.15.57:0/3139962085 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b4039840 con 0x7f82b4030660 2024-03-29T10:46:52.788 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.804+0000 7f82b9d9b640 1 -- 172.21.15.57:0/3139962085 learned_addr learned my addr 172.21.15.57:0/3139962085 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:46:52.788 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.804+0000 7f82bb59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:52.788 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.804+0000 7f82bb59e640 10 monclient(hunting): authenticate will time out at 1364.981845s 2024-03-29T10:46:52.790 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.805+0000 7f82b959a640 1 -- 172.21.15.57:0/3139962085 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609188355 0 0) 0x7f82b40c54c0 con 0x7f82b4038dc0 2024-03-29T10:46:52.790 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.805+0000 7f82b959a640 10 monclient(hunting): _negotiate 2024-03-29T10:46:52.790 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.805+0000 7f82b959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:52.790 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.805+0000 7f82b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:52.790 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.805+0000 7f82b959a640 10 monclient(hunting): my global_id is 4467 2024-03-29T10:46:52.790 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.805+0000 7f82b959a640 1 -- 172.21.15.57:0/3139962085 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82a4003520 con 0x7f82b4038dc0 2024-03-29T10:46:52.790 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.805+0000 7f82b959a640 1 -- 172.21.15.57:0/3139962085 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1272027134 0 0) 0x7f82b4039840 con 0x7f82b4030660 2024-03-29T10:46:52.790 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.805+0000 7f82b959a640 10 monclient(hunting): _negotiate 2024-03-29T10:46:52.790 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.805+0000 7f82b959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:52.790 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.805+0000 7f82b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:52.790 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.805+0000 7f82b959a640 10 monclient(hunting): my global_id is 4505 2024-03-29T10:46:52.791 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.805+0000 7f82b959a640 1 -- 172.21.15.57:0/3139962085 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b40c54c0 con 0x7f82b4030660 2024-03-29T10:46:52.791 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.806+0000 7f82b959a640 1 -- 172.21.15.57:0/3139962085 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2285605962 0 0) 0x7f82b40c54c0 con 0x7f82b4030660 2024-03-29T10:46:52.791 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.806+0000 7f82b959a640 1 -- 172.21.15.57:0/3139962085 >> v1:172.21.15.179:6789/0 conn(0x7f82b40389e0 legacy=0x7f82b4033520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:46:52.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.808+0000 7f82b959a640 1 -- 172.21.15.57:0/3139962085 >> v1:172.21.15.57:6789/0 conn(0x7f82b4038dc0 legacy=0x7f82b40cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:52.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.808+0000 7f82b959a640 10 monclient: _finish_hunting 0 2024-03-29T10:46:52.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.808+0000 7f82b959a640 1 monclient: found mon.noname-b 2024-03-29T10:46:52.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.808+0000 7f82b959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:52.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.808+0000 7f82b959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:46:52.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.808+0000 7f82b959a640 1 -- 172.21.15.57:0/3139962085 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b4035690 con 0x7f82b4030660 2024-03-29T10:46:52.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.808+0000 7f82b959a640 10 monclient: _finish_auth 0 2024-03-29T10:46:52.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.808+0000 7f82b959a640 10 monclient: _check_auth_tickets 2024-03-29T10:46:52.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.808+0000 7f82b959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:52.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.808+0000 7f82b959a640 1 -- 172.21.15.57:0/3139962085 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f82b0001520 con 0x7f82b4030660 2024-03-29T10:46:52.793 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.808+0000 7f82b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:52.794 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.808+0000 7f82b959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:46:52.794 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.808+0000 7f82b959a640 10 monclient: dump: 2024-03-29T10:46:52.794 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:52.794 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:52.794 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:52.794 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:52.794 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:52.794 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:52.794 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:52.794 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:52.794 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:52.794 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:52.794 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.808+0000 7f82b959a640 1 -- 172.21.15.57:0/3139962085 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82b0001700 con 0x7f82b4030660 2024-03-29T10:46:52.794 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.808+0000 7f82b959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:52.795 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.808+0000 7f82b959a640 1 -- 172.21.15.57:0/3139962085 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f82b0001b10 con 0x7f82b4030660 2024-03-29T10:46:52.795 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.808+0000 7f82b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:52.795 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.808+0000 7f82b959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:52.795 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.808+0000 7f82b959a640 10 monclient: dump: 2024-03-29T10:46:52.795 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:52.795 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:52.795 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:52.795 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:52.795 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:52.795 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:52.795 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:52.795 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:52.795 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:52.795 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:52.796 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.808+0000 7f82bb59e640 5 monclient: authenticate success, global_id 4505 2024-03-29T10:46:52.796 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.809+0000 7f82bb59e640 10 monclient: get_monmap_and_config success 2024-03-29T10:46:52.796 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.809+0000 7f82bb59e640 4 set_mon_vals no callback set 2024-03-29T10:46:52.796 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.809+0000 7f82bb59e640 10 monclient: shutdown 2024-03-29T10:46:52.796 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.810+0000 7f82bb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:52.796 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.810+0000 7f82bb59e640 1 -- 172.21.15.57:0/3139962085 >> v1:172.21.15.57:6790/0 conn(0x7f82b4030660 legacy=0x7f82b40d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:52.796 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.811+0000 7f82bb59e640 1 -- 172.21.15.57:0/3139962085 shutdown_connections 2024-03-29T10:46:52.797 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.811+0000 7f82bb59e640 1 -- 172.21.15.57:0/3139962085 >> 172.21.15.57:0/3139962085 conn(0x7f82b4005c00 msgr2=0x7f82b4006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:52.797 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.813+0000 7f82bb59e640 1 -- 172.21.15.57:0/3139962085 shutdown_connections 2024-03-29T10:46:52.799 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.813+0000 7f82bb59e640 1 -- 172.21.15.57:0/3139962085 wait complete. 2024-03-29T10:46:52.799 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.815+0000 7f82bb59e640 10 monclient: build_initial_monmap 2024-03-29T10:46:52.799 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.815+0000 7f82bb59e640 10 monclient: monmap: 2024-03-29T10:46:52.799 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:46:52.799 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:46:52.799 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:46:52.799 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:46:52.799 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:46:52.799 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:52.799 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:46:52.799 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:46:52.799 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:46:52.799 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:52.800 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.815+0000 7f82bb59e640 1 librados: starting msgr at 2024-03-29T10:46:52.800 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.815+0000 7f82bb59e640 1 librados: starting objecter 2024-03-29T10:46:52.800 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.815+0000 7f82bb59e640 1 Processor -- start 2024-03-29T10:46:52.800 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.815+0000 7f82bb59e640 1 -- start start 2024-03-29T10:46:52.800 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.816+0000 7f82bb59e640 1 librados: setting wanted keys 2024-03-29T10:46:52.801 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.816+0000 7f82bb59e640 1 librados: calling monclient init 2024-03-29T10:46:52.801 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.816+0000 7f82bb59e640 10 monclient: init 2024-03-29T10:46:52.801 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.816+0000 7f82bb59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:46:52.801 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.817+0000 7f82bb59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:46:52.802 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.817+0000 7f82bb59e640 10 monclient(hunting): picked mon.noname-a con 0x7f82b4030660 addr v1:172.21.15.57:6789/0 2024-03-29T10:46:52.806 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.820+0000 7f82bb59e640 10 monclient(hunting): picked mon.noname-b con 0x7f82b40389e0 addr v1:172.21.15.179:6789/0 2024-03-29T10:46:52.806 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.822+0000 7f82bb59e640 10 monclient(hunting): picked mon.noname-c con 0x7f82b40c7740 addr v1:172.21.15.57:6790/0 2024-03-29T10:46:52.806 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.822+0000 7f82bb59e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b40c2bb0 con 0x7f82b4030660 2024-03-29T10:46:52.806 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.822+0000 7f82bb59e640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b40c6f40 con 0x7f82b40389e0 2024-03-29T10:46:52.806 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.822+0000 7f82bb59e640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b40c7220 con 0x7f82b40c7740 2024-03-29T10:46:52.806 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.822+0000 7f82bb59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:46:52.806 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.822+0000 7f82bb59e640 10 monclient(hunting): authenticate will time out at 1364.999895s 2024-03-29T10:46:52.809 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.822+0000 7f82ba59c640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f82b4030660 0x7f82b415fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:42339/0 (socket says 172.21.15.57:42339) 2024-03-29T10:46:52.809 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.826+0000 7f82ba59c640 1 -- 172.21.15.57:0/13106446 learned_addr learned my addr 172.21.15.57:0/13106446 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:46:52.810 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.827+0000 7f82aaffd640 10 client.?.objecter ms_handle_connect 0x7f82b40c7740 2024-03-29T10:46:52.810 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.827+0000 7f82aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:52.810 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.827+0000 7f82aaffd640 10 client.?.objecter ms_handle_connect 0x7f82b40389e0 2024-03-29T10:46:52.810 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.827+0000 7f82aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:52.812 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.829+0000 7f82aaffd640 1 -- 172.21.15.57:0/13106446 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616651738 0 0) 0x7f82b40c6f40 con 0x7f82b40389e0 2024-03-29T10:46:52.812 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.829+0000 7f82aaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:52.812 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.829+0000 7f82aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:52.813 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.829+0000 7f82aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:52.813 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.829+0000 7f82aaffd640 10 monclient(hunting): my global_id is 4477 2024-03-29T10:46:52.813 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.829+0000 7f82aaffd640 1 -- 172.21.15.57:0/13106446 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f829c004eb0 con 0x7f82b40389e0 2024-03-29T10:46:52.813 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.829+0000 7f82aaffd640 1 -- 172.21.15.57:0/13106446 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709792165 0 0) 0x7f82b40c7220 con 0x7f82b40c7740 2024-03-29T10:46:52.813 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.829+0000 7f82aaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:46:52.813 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.829+0000 7f82aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:46:52.813 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.829+0000 7f82aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:46:52.813 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.829+0000 7f82aaffd640 10 monclient(hunting): my global_id is 4508 2024-03-29T10:46:52.813 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.829+0000 7f82aaffd640 1 -- 172.21.15.57:0/13106446 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b40c6f40 con 0x7f82b40c7740 2024-03-29T10:46:52.814 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.829+0000 7f82aaffd640 10 client.?.objecter ms_handle_connect 0x7f82b4030660 2024-03-29T10:46:52.814 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.829+0000 7f82aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:46:52.814 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.829+0000 7f82aaffd640 1 -- 172.21.15.57:0/13106446 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2031438785 0 0) 0x7f82b40c6f40 con 0x7f82b40c7740 2024-03-29T10:46:52.814 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.829+0000 7f82aaffd640 1 -- 172.21.15.57:0/13106446 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82b40c7220 con 0x7f82b40c7740 2024-03-29T10:46:52.814 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82aaffd640 1 -- 172.21.15.57:0/13106446 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f82b0001b10 con 0x7f82b40c7740 2024-03-29T10:46:52.814 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:52.814 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:46:52.814 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82aaffd640 10 monclient(hunting): dump: 2024-03-29T10:46:52.815 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:52.815 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:52.815 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:52.815 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:52.815 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:52.815 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:52.815 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:52.815 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:52.815 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:52.815 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:52.815 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:46:52.815 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82aaffd640 1 -- 172.21.15.57:0/13106446 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1033685930 0 0) 0x7f82b40c7220 con 0x7f82b40c7740 2024-03-29T10:46:52.816 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82aaffd640 1 -- 172.21.15.57:0/13106446 >> v1:172.21.15.179:6789/0 conn(0x7f82b40389e0 legacy=0x7f82b40c1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:52.816 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82aaffd640 1 -- 172.21.15.57:0/13106446 >> v1:172.21.15.57:6789/0 conn(0x7f82b4030660 legacy=0x7f82b415fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:52.816 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82aaffd640 10 monclient: _finish_hunting 0 2024-03-29T10:46:52.816 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82aaffd640 1 monclient: found mon.c 2024-03-29T10:46:52.816 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:46:52.816 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82aaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:46:52.816 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82aaffd640 1 -- 172.21.15.57:0/13106446 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b40c2e80 con 0x7f82b40c7740 2024-03-29T10:46:52.816 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82aaffd640 10 monclient: _finish_auth 0 2024-03-29T10:46:52.816 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82aaffd640 10 monclient: _check_auth_tickets 2024-03-29T10:46:52.816 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:46:52.816 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82bb59e640 5 monclient: authenticate success, global_id 4508 2024-03-29T10:46:52.816 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82bb59e640 10 monclient: _renew_subs 2024-03-29T10:46:52.817 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82bb59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:46:52.817 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82bb59e640 1 -- 172.21.15.57:0/13106446 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b4160460 con 0x7f82b40c7740 2024-03-29T10:46:52.817 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82bb59e640 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:46:52.817 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82bb59e640 10 monclient: _renew_subs 2024-03-29T10:46:52.817 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82bb59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:46:52.817 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82bb59e640 1 -- 172.21.15.57:0/13106446 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b402f680 con 0x7f82b40c7740 2024-03-29T10:46:52.817 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82bb59e640 1 librados: init done 2024-03-29T10:46:52.817 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82aaffd640 1 -- 172.21.15.57:0/13106446 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82b00038a0 con 0x7f82b40c7740 2024-03-29T10:46:52.817 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:46:52.817 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82aaffd640 1 -- 172.21.15.57:0/13106446 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f82b0003bb0 con 0x7f82b40c7740 2024-03-29T10:46:52.817 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.830+0000 7f82b8d99640 4 set_mon_vals no callback set 2024-03-29T10:46:52.817 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.831+0000 7f82aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:46:52.818 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.831+0000 7f82aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:46:52.818 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.831+0000 7f82aaffd640 10 monclient: dump: 2024-03-29T10:46:52.818 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:46:52.818 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:46:52.818 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:52.818 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:46:52.818 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:46:52.818 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:46:52.818 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:46:52.818 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:46:52.818 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:46:52.819 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:46:52.820 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.832+0000 7f82aaffd640 1 -- 172.21.15.57:0/13106446 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f82b0019b80 con 0x7f82b40c7740 2024-03-29T10:46:52.820 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.833+0000 7f82aaffd640 1 -- 172.21.15.57:0/13106446 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f82b00838f0 con 0x7f82b40c7740 2024-03-29T10:46:52.820 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.833+0000 7f82aaffd640 10 client.4508.objecter ms_dispatch 0x7f82bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:46:52.820 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.833+0000 7f82aaffd640 3 client.4508.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:46:52.820 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.833+0000 7f82aaffd640 3 client.4508.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:46:52.820 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.833+0000 7f82aaffd640 20 client.4508.objecter dump_active .. 0 homeless 2024-03-29T10:46:52.820 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.834+0000 7f82ba59c640 10 monclient: get_auth_request con 0x7f829c06dda0 auth_method 0 2024-03-29T10:46:52.821 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.835+0000 7f82bb59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:52.821 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.835+0000 7f82bb59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:52.821 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.835+0000 7f82bb59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:46:52.821 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.835+0000 7f82bb59e640 1 -- 172.21.15.57:0/13106446 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82a0005080 con 0x7f82b40c7740 2024-03-29T10:46:52.821 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.837+0000 7f82aaffd640 10 client.4508.objecter ms_handle_connect 0x7f829c06dda0 2024-03-29T10:46:52.829 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.846+0000 7f82aaffd640 1 -- 172.21.15.57:0/13106446 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f82b004f000 con 0x7f82b40c7740 2024-03-29T10:46:52.829 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.846+0000 7f82aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:46:52.829 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:52.846+0000 7f82aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:46:53.151 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:53.167+0000 7f82bb59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:53.151 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:53.167+0000 7f82bb59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:53.151 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:53.167+0000 7f82bb59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:46:53.151 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:53.167+0000 7f82bb59e640 1 -- 172.21.15.57:0/13106446 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82a0005330 con 0x7f82b40c7740 2024-03-29T10:46:53.153 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:53.170+0000 7f82aaffd640 1 -- 172.21.15.57:0/13106446 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f82b0051100 con 0x7f82b40c7740 2024-03-29T10:46:53.154 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:53.170+0000 7f82aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:46:53.154 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:53.170+0000 7f82aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:46:53.154 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:46:53.154 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:46:53.159 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:53.174+0000 7f82bb59e640 10 librados: watch_flush enter 2024-03-29T10:46:53.160 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:53.174+0000 7f82bb59e640 10 librados: watch_flush exit 2024-03-29T10:46:53.160 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:53.174+0000 7f82bb59e640 20 client.4508.objecter shutdown clearing up homeless session... 2024-03-29T10:46:53.160 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:53.174+0000 7f82bb59e640 10 client.4508.objecter successfully canceled tick 2024-03-29T10:46:53.160 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:53.174+0000 7f82bb59e640 1 -- 172.21.15.57:0/13106446 >> v1:172.21.15.179:6816/2698295936 conn(0x7f829c06dda0 legacy=0x7f829c070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:53.160 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:53.174+0000 7f82bb59e640 10 monclient: shutdown 2024-03-29T10:46:53.160 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:53.174+0000 7f82bb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:46:53.160 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:53.174+0000 7f82bb59e640 1 -- 172.21.15.57:0/13106446 >> v1:172.21.15.57:6790/0 conn(0x7f82b40c7740 legacy=0x7f82b40c24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:46:53.161 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:53.177+0000 7f82bb59e640 1 -- 172.21.15.57:0/13106446 shutdown_connections 2024-03-29T10:46:53.161 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:53.177+0000 7f82bb59e640 1 -- 172.21.15.57:0/13106446 >> 172.21.15.57:0/13106446 conn(0x7f82b4005c00 msgr2=0x7f82b40c4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:46:53.161 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:53.177+0000 7f82bb59e640 1 -- 172.21.15.57:0/13106446 shutdown_connections 2024-03-29T10:46:53.161 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:53.177+0000 7f82bb59e640 1 -- 172.21.15.57:0/13106446 wait complete. 2024-03-29T10:46:53.161 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:46:53.177+0000 7f82bb59e640 1 librados: shutdown 2024-03-29T10:46:53.177 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:46:59.985 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:00.133 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.150+0000 7fc4f1afb640 10 monclient: get_monmap_and_config 2024-03-29T10:47:00.133 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.150+0000 7fc4f1afb640 10 monclient: build_initial_monmap 2024-03-29T10:47:00.134 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.150+0000 7fc4f1afb640 10 monclient: monmap: 2024-03-29T10:47:00.134 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:47:00.134 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:00.134 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:47:00.152049+0000 2024-03-29T10:47:00.134 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:47:00.152049+0000 2024-03-29T10:47:00.134 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:00.134 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:00.134 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:47:00.134 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:47:00.134 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:47:00.134 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:00.135 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.151+0000 7fc4f1afb640 1 Processor -- start 2024-03-29T10:47:00.136 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.151+0000 7fc4f1afb640 1 -- start start 2024-03-29T10:47:00.136 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.151+0000 7fc4f1afb640 10 monclient: init 2024-03-29T10:47:00.136 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.152+0000 7fc4f1afb640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:00.136 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.152+0000 7fc4f1afb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:00.136 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.152+0000 7fc4f1afb640 10 monclient(hunting): picked mon.noname-a con 0x7fc4ec1457f0 addr v1:172.21.15.57:6789/0 2024-03-29T10:47:00.136 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.152+0000 7fc4f1afb640 10 monclient(hunting): picked mon.noname-b con 0x7fc4ec13f650 addr v1:172.21.15.57:6790/0 2024-03-29T10:47:00.136 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.152+0000 7fc4f1afb640 10 monclient(hunting): picked mon.noname-c con 0x7fc4ec0a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T10:47:00.136 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.152+0000 7fc4f1afb640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ec0aaef0 con 0x7fc4ec1457f0 2024-03-29T10:47:00.136 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.152+0000 7fc4f1afb640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ec145bd0 con 0x7fc4ec0a9d70 2024-03-29T10:47:00.136 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.152+0000 7fc4f1afb640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ec145eb0 con 0x7fc4ec13f650 2024-03-29T10:47:00.136 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.152+0000 7fc4f1afb640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:00.136 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.152+0000 7fc4f1afb640 10 monclient(hunting): authenticate will time out at 1372.329411s 2024-03-29T10:47:00.136 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.152+0000 7fc4eb7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fc4ec1457f0 0x7fc4ec13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:37170/0 (socket says 172.21.15.57:37170) 2024-03-29T10:47:00.136 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.152+0000 7fc4eb7fe640 1 -- 172.21.15.57:0/990385179 learned_addr learned my addr 172.21.15.57:0/990385179 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:47:00.139 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.156+0000 7fc4ea7fc640 1 -- 172.21.15.57:0/990385179 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928874083 0 0) 0x7fc4ec0aaef0 con 0x7fc4ec1457f0 2024-03-29T10:47:00.139 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.156+0000 7fc4ea7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:00.139 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.156+0000 7fc4ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:00.139 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.156+0000 7fc4ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:00.140 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.156+0000 7fc4ea7fc640 10 monclient(hunting): my global_id is 4470 2024-03-29T10:47:00.140 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.156+0000 7fc4ea7fc640 1 -- 172.21.15.57:0/990385179 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4d4003520 con 0x7fc4ec1457f0 2024-03-29T10:47:00.140 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.156+0000 7fc4eb7fe640 1 -- 172.21.15.57:0/990385179 >> v1:172.21.15.57:6789/0 conn(0x7fc4ec1457f0 legacy=0x7fc4ec13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-29T10:47:00.140 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.156+0000 7fc4eb7fe640 1 -- 172.21.15.57:0/990385179 >> v1:172.21.15.57:6789/0 conn(0x7fc4ec1457f0 legacy=0x7fc4ec13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:47:00.140 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.156+0000 7fc4eb7fe640 1 --1- 172.21.15.57:0/990385179 >> v1:172.21.15.57:6789/0 conn(0x7fc4ec1457f0 0x7fc4ec13ef40 :-1 s=OPENED pgs=715754 cs=1 l=1).handle_message read tag failed 2024-03-29T10:47:00.140 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.156+0000 7fc4eb7fe640 1 --1- 172.21.15.57:0/990385179 >> v1:172.21.15.57:6789/0 conn(0x7fc4ec1457f0 0x7fc4ec13ef40 :-1 s=OPENED pgs=715754 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:47:00.141 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.156+0000 7fc4ea7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.57:6789/0 2024-03-29T10:47:00.144 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.160+0000 7fc4ea7fc640 1 -- 172.21.15.57:0/990385179 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840146927 0 0) 0x7fc4ec145eb0 con 0x7fc4ec13f650 2024-03-29T10:47:00.144 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.160+0000 7fc4ea7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:47:00.144 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.160+0000 7fc4ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:00.144 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.160+0000 7fc4ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:00.144 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.160+0000 7fc4ea7fc640 10 monclient(hunting): my global_id is 4511 2024-03-29T10:47:00.144 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.161+0000 7fc4ea7fc640 1 -- 172.21.15.57:0/990385179 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4ec0aaef0 con 0x7fc4ec13f650 2024-03-29T10:47:00.146 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.162+0000 7fc4ea7fc640 1 -- 172.21.15.57:0/990385179 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 692324022 0 0) 0x7fc4ec0aaef0 con 0x7fc4ec13f650 2024-03-29T10:47:00.146 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.162+0000 7fc4ea7fc640 1 -- 172.21.15.57:0/990385179 >> v1:172.21.15.179:6789/0 conn(0x7fc4ec0a9d70 legacy=0x7fc4ec14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:00.146 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.162+0000 7fc4ea7fc640 1 -- 172.21.15.57:0/990385179 >> v1:172.21.15.57:6789/0 conn(0x7fc4ec1457f0 legacy=0x7fc4ec13ef40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:47:00.146 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.162+0000 7fc4ea7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:47:00.147 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.162+0000 7fc4ea7fc640 1 monclient: found mon.noname-b 2024-03-29T10:47:00.147 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.162+0000 7fc4ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:00.147 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.162+0000 7fc4ea7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:47:00.147 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.162+0000 7fc4ea7fc640 1 -- 172.21.15.57:0/990385179 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ec146340 con 0x7fc4ec13f650 2024-03-29T10:47:00.147 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.162+0000 7fc4ea7fc640 10 monclient: _finish_auth 0 2024-03-29T10:47:00.147 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.162+0000 7fc4ea7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:47:00.147 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.162+0000 7fc4ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:00.147 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.162+0000 7fc4ea7fc640 1 -- 172.21.15.57:0/990385179 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fc4dc001670 con 0x7fc4ec13f650 2024-03-29T10:47:00.147 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.162+0000 7fc4ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:00.147 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.162+0000 7fc4ea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:00.147 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.162+0000 7fc4ea7fc640 10 monclient: dump: 2024-03-29T10:47:00.147 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:00.148 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:00.148 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:00.148 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:00.148 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:00.148 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:00.148 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:00.148 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:00.148 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:00.148 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:00.148 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.163+0000 7fc4f1afb640 5 monclient: authenticate success, global_id 4511 2024-03-29T10:47:00.149 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.163+0000 7fc4f1afb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:00.149 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.165+0000 7fc4ea7fc640 1 -- 172.21.15.57:0/990385179 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4dc0017d0 con 0x7fc4ec13f650 2024-03-29T10:47:00.149 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.165+0000 7fc4ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:00.149 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.165+0000 7fc4ea7fc640 1 -- 172.21.15.57:0/990385179 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fc4dc001c20 con 0x7fc4ec13f650 2024-03-29T10:47:00.149 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.165+0000 7fc4ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:00.149 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.165+0000 7fc4ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:00.150 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.165+0000 7fc4ea7fc640 10 monclient: dump: 2024-03-29T10:47:00.150 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:00.150 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:00.150 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:00.150 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:00.150 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:00.150 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:00.150 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:00.150 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:00.150 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:00.150 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:00.150 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.165+0000 7fc4f1afb640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:00.151 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.165+0000 7fc4f1afb640 4 set_mon_vals no callback set 2024-03-29T10:47:00.152 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.166+0000 7fc4f1afb640 10 monclient: shutdown 2024-03-29T10:47:00.152 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.166+0000 7fc4f1afb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:00.152 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.166+0000 7fc4f1afb640 1 -- 172.21.15.57:0/990385179 >> v1:172.21.15.57:6790/0 conn(0x7fc4ec13f650 legacy=0x7fc4ec0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:00.152 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.166+0000 7fc4f1afb640 1 -- 172.21.15.57:0/990385179 shutdown_connections 2024-03-29T10:47:00.152 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.166+0000 7fc4f1afb640 1 -- 172.21.15.57:0/990385179 >> 172.21.15.57:0/990385179 conn(0x7fc4ec066a00 msgr2=0x7fc4ec066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:00.152 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.167+0000 7fc4f1afb640 1 -- 172.21.15.57:0/990385179 shutdown_connections 2024-03-29T10:47:00.152 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.167+0000 7fc4f1afb640 1 -- 172.21.15.57:0/990385179 wait complete. 2024-03-29T10:47:00.152 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.168+0000 7fc4f1afb640 10 monclient: build_initial_monmap 2024-03-29T10:47:00.152 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.168+0000 7fc4f1afb640 10 monclient: monmap: 2024-03-29T10:47:00.152 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:47:00.152 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:00.152 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:47:00.152 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:47:00.153 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:00.153 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:00.153 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:47:00.153 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:47:00.153 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:47:00.153 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:00.153 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.169+0000 7fc4f1afb640 1 librados: starting msgr at 2024-03-29T10:47:00.153 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.169+0000 7fc4f1afb640 1 librados: starting objecter 2024-03-29T10:47:00.153 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.169+0000 7fc4f1afb640 1 Processor -- start 2024-03-29T10:47:00.156 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.169+0000 7fc4f1afb640 1 -- start start 2024-03-29T10:47:00.156 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.169+0000 7fc4f1afb640 1 librados: setting wanted keys 2024-03-29T10:47:00.156 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.169+0000 7fc4f1afb640 1 librados: calling monclient init 2024-03-29T10:47:00.156 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.169+0000 7fc4f1afb640 10 monclient: init 2024-03-29T10:47:00.156 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.170+0000 7fc4f1afb640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:00.156 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.170+0000 7fc4f1afb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:00.156 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.170+0000 7fc4f1afb640 10 monclient(hunting): picked mon.noname-b con 0x7fc4ec0a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T10:47:00.156 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.170+0000 7fc4f1afb640 10 monclient(hunting): picked mon.noname-c con 0x7fc4ec1457f0 addr v1:172.21.15.57:6790/0 2024-03-29T10:47:00.156 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.170+0000 7fc4f1afb640 10 monclient(hunting): picked mon.noname-a con 0x7fc4ec143810 addr v1:172.21.15.57:6789/0 2024-03-29T10:47:00.156 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.170+0000 7fc4f1afb640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ec1e42a0 con 0x7fc4ec143810 2024-03-29T10:47:00.156 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.170+0000 7fc4f1afb640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ec139990 con 0x7fc4ec0a9d70 2024-03-29T10:47:00.157 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.170+0000 7fc4f1afb640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ec13fda0 con 0x7fc4ec1457f0 2024-03-29T10:47:00.157 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.170+0000 7fc4f1afb640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:00.157 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.170+0000 7fc4f1afb640 10 monclient(hunting): authenticate will time out at 1372.347477s 2024-03-29T10:47:00.157 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.174+0000 7fc4eb7fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fc4ec0a9d70 0x7fc4ec1e4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:47303/0 (socket says 172.21.15.57:47303) 2024-03-29T10:47:00.157 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.174+0000 7fc4eb7fe640 1 -- 172.21.15.57:0/3644292948 learned_addr learned my addr 172.21.15.57:0/3644292948 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:47:00.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.174+0000 7fc4d3fff640 10 client.?.objecter ms_handle_connect 0x7fc4ec0a9d70 2024-03-29T10:47:00.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.174+0000 7fc4d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:00.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.174+0000 7fc4d3fff640 1 -- 172.21.15.57:0/3644292948 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2583810548 0 0) 0x7fc4ec139990 con 0x7fc4ec0a9d70 2024-03-29T10:47:00.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.174+0000 7fc4d3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:47:00.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.174+0000 7fc4d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:00.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.174+0000 7fc4d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:00.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.174+0000 7fc4d3fff640 10 monclient(hunting): my global_id is 4483 2024-03-29T10:47:00.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.174+0000 7fc4d3fff640 1 -- 172.21.15.57:0/3644292948 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4d8004eb0 con 0x7fc4ec0a9d70 2024-03-29T10:47:00.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.175+0000 7fc4d3fff640 1 -- 172.21.15.57:0/3644292948 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2686634015 0 0) 0x7fc4d8004eb0 con 0x7fc4ec0a9d70 2024-03-29T10:47:00.158 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.175+0000 7fc4d3fff640 1 -- 172.21.15.57:0/3644292948 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4ec139990 con 0x7fc4ec0a9d70 2024-03-29T10:47:00.159 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.175+0000 7fc4d3fff640 1 -- 172.21.15.57:0/3644292948 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fc4e4001c70 con 0x7fc4ec0a9d70 2024-03-29T10:47:00.159 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.175+0000 7fc4d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:00.159 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.175+0000 7fc4d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:00.159 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.175+0000 7fc4d3fff640 10 monclient(hunting): dump: 2024-03-29T10:47:00.159 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:00.159 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:00.159 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:00.159 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:00.159 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:00.159 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:00.159 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:00.159 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:00.159 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:00.159 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:00.160 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.175+0000 7fc4d3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:00.160 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.176+0000 7fc4d3fff640 10 client.?.objecter ms_handle_connect 0x7fc4ec143810 2024-03-29T10:47:00.160 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.176+0000 7fc4d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:00.160 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.176+0000 7fc4d3fff640 10 client.?.objecter ms_handle_connect 0x7fc4ec1457f0 2024-03-29T10:47:00.160 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.176+0000 7fc4d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:00.160 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.176+0000 7fc4d3fff640 1 -- 172.21.15.57:0/3644292948 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1675045451 0 0) 0x7fc4ec139990 con 0x7fc4ec0a9d70 2024-03-29T10:47:00.161 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.176+0000 7fc4d3fff640 1 -- 172.21.15.57:0/3644292948 >> v1:172.21.15.57:6790/0 conn(0x7fc4ec1457f0 legacy=0x7fc4ec13a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:00.161 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.176+0000 7fc4d3fff640 1 -- 172.21.15.57:0/3644292948 >> v1:172.21.15.57:6789/0 conn(0x7fc4ec143810 legacy=0x7fc4ec1d6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:00.161 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.176+0000 7fc4d3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:47:00.161 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.176+0000 7fc4d3fff640 1 monclient: found mon.b 2024-03-29T10:47:00.161 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.176+0000 7fc4d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:00.161 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.176+0000 7fc4d3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:47:00.161 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.176+0000 7fc4d3fff640 1 -- 172.21.15.57:0/3644292948 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ec1e4e40 con 0x7fc4ec0a9d70 2024-03-29T10:47:00.161 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.176+0000 7fc4d3fff640 10 monclient: _finish_auth 0 2024-03-29T10:47:00.161 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.176+0000 7fc4d3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:47:00.161 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.176+0000 7fc4d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:00.161 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.176+0000 7fc4f1afb640 5 monclient: authenticate success, global_id 4483 2024-03-29T10:47:00.161 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.176+0000 7fc4f1afb640 10 monclient: _renew_subs 2024-03-29T10:47:00.161 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.176+0000 7fc4f1afb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:47:00.161 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.176+0000 7fc4f1afb640 1 -- 172.21.15.57:0/3644292948 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4ec13ab10 con 0x7fc4ec0a9d70 2024-03-29T10:47:00.161 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.176+0000 7fc4f1afb640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:00.162 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.176+0000 7fc4f1afb640 10 monclient: _renew_subs 2024-03-29T10:47:00.162 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.176+0000 7fc4f1afb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:47:00.162 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.176+0000 7fc4f1afb640 1 -- 172.21.15.57:0/3644292948 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4ec1d7230 con 0x7fc4ec0a9d70 2024-03-29T10:47:00.162 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.176+0000 7fc4f1afb640 1 librados: init done 2024-03-29T10:47:00.163 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.178+0000 7fc4d3fff640 1 -- 172.21.15.57:0/3644292948 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4e4004b60 con 0x7fc4ec0a9d70 2024-03-29T10:47:00.163 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.178+0000 7fc4d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:00.163 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.178+0000 7fc4d3fff640 1 -- 172.21.15.57:0/3644292948 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fc4e4004f70 con 0x7fc4ec0a9d70 2024-03-29T10:47:00.163 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.178+0000 7fc4d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:00.163 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.178+0000 7fc4d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:00.163 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.178+0000 7fc4d3fff640 10 monclient: dump: 2024-03-29T10:47:00.163 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:00.163 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:00.163 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:00.163 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:00.163 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:00.164 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:00.164 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:00.164 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:00.164 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:00.164 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:00.164 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.178+0000 7fc4f1afb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:00.164 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.178+0000 7fc4f1afb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:00.164 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.178+0000 7fc4f1afb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:47:00.164 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.178+0000 7fc4f1afb640 1 -- 172.21.15.57:0/3644292948 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4ec1e4070 con 0x7fc4ec0a9d70 2024-03-29T10:47:00.166 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.179+0000 7fc4e9ffb640 4 set_mon_vals no callback set 2024-03-29T10:47:00.166 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.180+0000 7fc4d3fff640 1 -- 172.21.15.57:0/3644292948 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fc4e40050d0 con 0x7fc4ec0a9d70 2024-03-29T10:47:00.166 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.181+0000 7fc4d3fff640 1 -- 172.21.15.57:0/3644292948 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7fc4e4084c50 con 0x7fc4ec0a9d70 2024-03-29T10:47:00.166 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.181+0000 7fc4d3fff640 10 client.4483.objecter ms_dispatch 0x7fc4ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:00.166 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.181+0000 7fc4d3fff640 3 client.4483.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:00.166 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.181+0000 7fc4d3fff640 3 client.4483.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:00.166 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.181+0000 7fc4d3fff640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-29T10:47:00.166 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.183+0000 7fc4eaffd640 10 monclient: get_auth_request con 0x7fc4d806dcd0 auth_method 0 2024-03-29T10:47:00.167 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.183+0000 7fc4d3fff640 10 client.4483.objecter ms_handle_connect 0x7fc4d806dcd0 2024-03-29T10:47:00.171 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.187+0000 7fc4d3fff640 1 -- 172.21.15.57:0/3644292948 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc4e4050530 con 0x7fc4ec0a9d70 2024-03-29T10:47:00.171 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.187+0000 7fc4d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:00.171 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.187+0000 7fc4d3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:00.513 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.530+0000 7fc4f1afb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:00.514 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.530+0000 7fc4f1afb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:00.514 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.530+0000 7fc4f1afb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:47:00.514 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.530+0000 7fc4f1afb640 1 -- 172.21.15.57:0/3644292948 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4ec1e4fd0 con 0x7fc4ec0a9d70 2024-03-29T10:47:00.516 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.533+0000 7fc4d3fff640 1 -- 172.21.15.57:0/3644292948 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7fc4e4052460 con 0x7fc4ec0a9d70 2024-03-29T10:47:00.516 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.533+0000 7fc4d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:00.516 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.533+0000 7fc4d3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:00.517 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:47:00.517 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:00.523 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.537+0000 7fc4f1afb640 10 librados: watch_flush enter 2024-03-29T10:47:00.523 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.537+0000 7fc4f1afb640 10 librados: watch_flush exit 2024-03-29T10:47:00.523 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.537+0000 7fc4f1afb640 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-29T10:47:00.523 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.537+0000 7fc4f1afb640 10 client.4483.objecter successfully canceled tick 2024-03-29T10:47:00.523 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.538+0000 7fc4f1afb640 1 -- 172.21.15.57:0/3644292948 >> v1:172.21.15.179:6816/2698295936 conn(0x7fc4d806dcd0 legacy=0x7fc4d8070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:00.524 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.538+0000 7fc4f1afb640 10 monclient: shutdown 2024-03-29T10:47:00.524 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.538+0000 7fc4f1afb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:00.524 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.538+0000 7fc4f1afb640 1 -- 172.21.15.57:0/3644292948 >> v1:172.21.15.179:6789/0 conn(0x7fc4ec0a9d70 legacy=0x7fc4ec1e4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:00.524 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.538+0000 7fc4f1afb640 1 -- 172.21.15.57:0/3644292948 shutdown_connections 2024-03-29T10:47:00.524 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.538+0000 7fc4f1afb640 1 -- 172.21.15.57:0/3644292948 >> 172.21.15.57:0/3644292948 conn(0x7fc4ec066a00 msgr2=0x7fc4ec0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:00.524 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.538+0000 7fc4f1afb640 1 -- 172.21.15.57:0/3644292948 shutdown_connections 2024-03-29T10:47:00.525 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.538+0000 7fc4f1afb640 1 -- 172.21.15.57:0/3644292948 wait complete. 2024-03-29T10:47:00.525 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:00.538+0000 7fc4f1afb640 1 librados: shutdown 2024-03-29T10:47:00.538 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:07.345 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:07.483 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.501+0000 7fe39128d640 10 monclient: get_monmap_and_config 2024-03-29T10:47:07.483 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.501+0000 7fe39128d640 10 monclient: build_initial_monmap 2024-03-29T10:47:07.484 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.501+0000 7fe39128d640 10 monclient: monmap: 2024-03-29T10:47:07.484 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:47:07.484 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:07.484 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:47:07.502038+0000 2024-03-29T10:47:07.484 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:47:07.502038+0000 2024-03-29T10:47:07.484 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:07.484 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:07.484 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:47:07.484 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:47:07.484 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:47:07.484 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:07.486 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.502+0000 7fe39128d640 1 Processor -- start 2024-03-29T10:47:07.486 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.502+0000 7fe39128d640 1 -- start start 2024-03-29T10:47:07.486 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.502+0000 7fe39128d640 10 monclient: init 2024-03-29T10:47:07.486 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.503+0000 7fe39128d640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:07.486 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.503+0000 7fe39128d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:47:07.486 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.503+0000 7fe39128d640 10 monclient(hunting): picked mon.noname-b con 0x7fe38c0b1f60 addr v1:172.21.15.57:6790/0 2024-03-29T10:47:07.486 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.503+0000 7fe39128d640 10 monclient(hunting): picked mon.noname-c con 0x7fe38c071eb0 addr v1:172.21.15.179:6789/0 2024-03-29T10:47:07.486 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.503+0000 7fe39128d640 10 monclient(hunting): picked mon.noname-a con 0x7fe38c143d10 addr v1:172.21.15.57:6789/0 2024-03-29T10:47:07.486 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.503+0000 7fe39128d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe38c06ef80 con 0x7fe38c143d10 2024-03-29T10:47:07.487 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.503+0000 7fe39128d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe38c06f260 con 0x7fe38c071eb0 2024-03-29T10:47:07.487 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.503+0000 7fe39128d640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe38c0b40c0 con 0x7fe38c0b1f60 2024-03-29T10:47:07.487 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.503+0000 7fe39128d640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:07.487 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.503+0000 7fe39128d640 10 monclient(hunting): authenticate will time out at 1379.679223s 2024-03-29T10:47:07.488 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.506+0000 7fe38affd640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7fe38c0b1f60 0x7fe38c143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:51896/0 (socket says 172.21.15.57:51896) 2024-03-29T10:47:07.488 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.506+0000 7fe38affd640 1 -- 172.21.15.57:0/351964865 learned_addr learned my addr 172.21.15.57:0/351964865 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:47:07.494 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.507+0000 7fe389ffb640 1 -- 172.21.15.57:0/351964865 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2362712660 0 0) 0x7fe38c0b40c0 con 0x7fe38c0b1f60 2024-03-29T10:47:07.494 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.507+0000 7fe389ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:07.494 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.507+0000 7fe389ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:07.494 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.507+0000 7fe389ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:07.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.507+0000 7fe389ffb640 10 monclient(hunting): my global_id is 4517 2024-03-29T10:47:07.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.507+0000 7fe389ffb640 1 -- 172.21.15.57:0/351964865 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe374003520 con 0x7fe38c0b1f60 2024-03-29T10:47:07.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.508+0000 7fe389ffb640 1 -- 172.21.15.57:0/351964865 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2697962244 0 0) 0x7fe38c06f260 con 0x7fe38c071eb0 2024-03-29T10:47:07.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.508+0000 7fe389ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:07.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.508+0000 7fe389ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:07.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.508+0000 7fe389ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:07.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.508+0000 7fe389ffb640 10 monclient(hunting): my global_id is 4486 2024-03-29T10:47:07.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.508+0000 7fe389ffb640 1 -- 172.21.15.57:0/351964865 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe38c0b40c0 con 0x7fe38c071eb0 2024-03-29T10:47:07.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.508+0000 7fe389ffb640 1 -- 172.21.15.57:0/351964865 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3608449802 0 0) 0x7fe374003520 con 0x7fe38c0b1f60 2024-03-29T10:47:07.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.508+0000 7fe389ffb640 1 -- 172.21.15.57:0/351964865 >> v1:172.21.15.179:6789/0 conn(0x7fe38c071eb0 legacy=0x7fe38c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:07.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.508+0000 7fe389ffb640 1 -- 172.21.15.57:0/351964865 >> v1:172.21.15.57:6789/0 conn(0x7fe38c143d10 legacy=0x7fe38c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:47:07.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.508+0000 7fe389ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:07.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.508+0000 7fe389ffb640 1 monclient: found mon.noname-b 2024-03-29T10:47:07.495 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.508+0000 7fe389ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:07.496 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.508+0000 7fe389ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:47:07.496 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.508+0000 7fe389ffb640 1 -- 172.21.15.57:0/351964865 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe38c155d70 con 0x7fe38c0b1f60 2024-03-29T10:47:07.496 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.508+0000 7fe389ffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:07.496 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.508+0000 7fe389ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:07.496 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.508+0000 7fe389ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:07.496 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.508+0000 7fe389ffb640 1 -- 172.21.15.57:0/351964865 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fe384001a10 con 0x7fe38c0b1f60 2024-03-29T10:47:07.496 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.508+0000 7fe389ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:07.496 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.508+0000 7fe389ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:07.496 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.508+0000 7fe389ffb640 10 monclient: dump: 2024-03-29T10:47:07.496 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:07.496 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:07.496 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:07.496 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:07.496 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:07.496 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:07.497 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:07.497 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:07.497 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:07.497 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:07.497 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.509+0000 7fe389ffb640 1 -- 172.21.15.57:0/351964865 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe384001c30 con 0x7fe38c0b1f60 2024-03-29T10:47:07.497 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.509+0000 7fe389ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:07.497 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.509+0000 7fe389ffb640 1 -- 172.21.15.57:0/351964865 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fe384001fc0 con 0x7fe38c0b1f60 2024-03-29T10:47:07.497 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.509+0000 7fe389ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:07.497 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.509+0000 7fe389ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:07.497 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.509+0000 7fe389ffb640 10 monclient: dump: 2024-03-29T10:47:07.497 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:07.497 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:07.497 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:07.497 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:07.497 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:07.498 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:07.498 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:07.498 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:07.498 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:07.498 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:07.498 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.509+0000 7fe39128d640 5 monclient: authenticate success, global_id 4517 2024-03-29T10:47:07.498 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.509+0000 7fe39128d640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:07.498 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.509+0000 7fe39128d640 4 set_mon_vals no callback set 2024-03-29T10:47:07.498 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.510+0000 7fe39128d640 10 monclient: shutdown 2024-03-29T10:47:07.498 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.510+0000 7fe39128d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:07.498 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.510+0000 7fe39128d640 1 -- 172.21.15.57:0/351964865 >> v1:172.21.15.57:6790/0 conn(0x7fe38c0b1f60 legacy=0x7fe38c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:07.498 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.510+0000 7fe39128d640 1 -- 172.21.15.57:0/351964865 shutdown_connections 2024-03-29T10:47:07.498 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.510+0000 7fe39128d640 1 -- 172.21.15.57:0/351964865 >> 172.21.15.57:0/351964865 conn(0x7fe38c066a00 msgr2=0x7fe38c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:07.498 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.511+0000 7fe39128d640 1 -- 172.21.15.57:0/351964865 shutdown_connections 2024-03-29T10:47:07.499 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.512+0000 7fe39128d640 1 -- 172.21.15.57:0/351964865 wait complete. 2024-03-29T10:47:07.499 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.513+0000 7fe39128d640 10 monclient: build_initial_monmap 2024-03-29T10:47:07.499 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.513+0000 7fe39128d640 10 monclient: monmap: 2024-03-29T10:47:07.500 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:47:07.500 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:07.500 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:47:07.500 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:47:07.500 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:07.500 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:07.500 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:47:07.500 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:47:07.500 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:47:07.500 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:07.500 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.513+0000 7fe39128d640 1 librados: starting msgr at 2024-03-29T10:47:07.500 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.513+0000 7fe39128d640 1 librados: starting objecter 2024-03-29T10:47:07.500 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.513+0000 7fe39128d640 1 Processor -- start 2024-03-29T10:47:07.500 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.513+0000 7fe39128d640 1 -- start start 2024-03-29T10:47:07.501 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.513+0000 7fe39128d640 1 librados: setting wanted keys 2024-03-29T10:47:07.501 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.513+0000 7fe39128d640 1 librados: calling monclient init 2024-03-29T10:47:07.501 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.513+0000 7fe39128d640 10 monclient: init 2024-03-29T10:47:07.501 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.514+0000 7fe39128d640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:07.501 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.514+0000 7fe39128d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:47:07.501 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.514+0000 7fe39128d640 10 monclient(hunting): picked mon.noname-c con 0x7fe38c0b1f60 addr v1:172.21.15.57:6790/0 2024-03-29T10:47:07.501 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.514+0000 7fe39128d640 10 monclient(hunting): picked mon.noname-b con 0x7fe38c143d10 addr v1:172.21.15.179:6789/0 2024-03-29T10:47:07.501 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.514+0000 7fe39128d640 10 monclient(hunting): picked mon.noname-a con 0x7fe38c149380 addr v1:172.21.15.57:6789/0 2024-03-29T10:47:07.501 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.514+0000 7fe39128d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe38c13fd40 con 0x7fe38c149380 2024-03-29T10:47:07.501 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.514+0000 7fe39128d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe38c14afb0 con 0x7fe38c143d10 2024-03-29T10:47:07.501 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.514+0000 7fe39128d640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe38c13b9d0 con 0x7fe38c0b1f60 2024-03-29T10:47:07.501 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.514+0000 7fe39128d640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:07.501 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.514+0000 7fe39128d640 10 monclient(hunting): authenticate will time out at 1379.690633s 2024-03-29T10:47:07.501 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.516+0000 7fe38affd640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7fe38c0b1f60 0x7fe38c13f630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:52374/0 (socket says 172.21.15.57:52374) 2024-03-29T10:47:07.501 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.516+0000 7fe38affd640 1 -- 172.21.15.57:0/900243343 learned_addr learned my addr 172.21.15.57:0/900243343 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:47:07.502 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.517+0000 7fe3737fe640 10 client.?.objecter ms_handle_connect 0x7fe38c0b1f60 2024-03-29T10:47:07.502 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.517+0000 7fe3737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:07.502 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.517+0000 7fe3737fe640 1 -- 172.21.15.57:0/900243343 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277737147 0 0) 0x7fe38c13b9d0 con 0x7fe38c0b1f60 2024-03-29T10:47:07.502 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.517+0000 7fe3737fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:07.502 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.517+0000 7fe3737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:07.502 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.517+0000 7fe3737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:07.502 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.517+0000 7fe3737fe640 10 monclient(hunting): my global_id is 4520 2024-03-29T10:47:07.502 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.517+0000 7fe3737fe640 1 -- 172.21.15.57:0/900243343 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe378004eb0 con 0x7fe38c0b1f60 2024-03-29T10:47:07.506 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.518+0000 7fe3737fe640 1 -- 172.21.15.57:0/900243343 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4181671439 0 0) 0x7fe378004eb0 con 0x7fe38c0b1f60 2024-03-29T10:47:07.506 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.518+0000 7fe3737fe640 1 -- 172.21.15.57:0/900243343 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe38c13b9d0 con 0x7fe38c0b1f60 2024-03-29T10:47:07.506 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.518+0000 7fe3737fe640 1 -- 172.21.15.57:0/900243343 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fe384001a10 con 0x7fe38c0b1f60 2024-03-29T10:47:07.506 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.518+0000 7fe3737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:07.506 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.518+0000 7fe3737fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:07.506 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.518+0000 7fe3737fe640 10 monclient(hunting): dump: 2024-03-29T10:47:07.506 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:07.506 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:07.506 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:07.507 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:07.507 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:07.507 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:07.507 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:07.507 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:07.507 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:07.507 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:07.507 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.518+0000 7fe3737fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:07.507 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.518+0000 7fe3737fe640 10 client.?.objecter ms_handle_connect 0x7fe38c149380 2024-03-29T10:47:07.507 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.518+0000 7fe3737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:07.507 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.518+0000 7fe3737fe640 1 -- 172.21.15.57:0/900243343 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3382337601 0 0) 0x7fe38c13fd40 con 0x7fe38c149380 2024-03-29T10:47:07.507 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.518+0000 7fe3737fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:07.507 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.518+0000 7fe3737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:07.507 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.518+0000 7fe3737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:07.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.518+0000 7fe3737fe640 10 monclient(hunting): my global_id is 4476 2024-03-29T10:47:07.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.518+0000 7fe3737fe640 1 -- 172.21.15.57:0/900243343 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe378004eb0 con 0x7fe38c149380 2024-03-29T10:47:07.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.519+0000 7fe3737fe640 1 -- 172.21.15.57:0/900243343 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3140402452 0 0) 0x7fe38c13b9d0 con 0x7fe38c0b1f60 2024-03-29T10:47:07.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.519+0000 7fe3737fe640 1 -- 172.21.15.57:0/900243343 >> v1:172.21.15.179:6789/0 conn(0x7fe38c143d10 legacy=0x7fe38c13abb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:47:07.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.519+0000 7fe3737fe640 1 -- 172.21.15.57:0/900243343 >> v1:172.21.15.57:6789/0 conn(0x7fe38c149380 legacy=0x7fe38c13b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:07.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.519+0000 7fe3737fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:07.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.519+0000 7fe3737fe640 1 monclient: found mon.c 2024-03-29T10:47:07.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.519+0000 7fe3737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:07.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.519+0000 7fe3737fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:47:07.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.519+0000 7fe3737fe640 1 -- 172.21.15.57:0/900243343 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe38c14b220 con 0x7fe38c0b1f60 2024-03-29T10:47:07.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.519+0000 7fe3737fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:07.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.519+0000 7fe3737fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:07.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.519+0000 7fe3737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:07.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.519+0000 7fe39128d640 5 monclient: authenticate success, global_id 4520 2024-03-29T10:47:07.508 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.519+0000 7fe39128d640 10 monclient: _renew_subs 2024-03-29T10:47:07.509 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.519+0000 7fe39128d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:47:07.509 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.519+0000 7fe39128d640 1 -- 172.21.15.57:0/900243343 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe38c14b500 con 0x7fe38c0b1f60 2024-03-29T10:47:07.509 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.519+0000 7fe39128d640 10 client.4520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:07.509 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.519+0000 7fe39128d640 10 monclient: _renew_subs 2024-03-29T10:47:07.509 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.519+0000 7fe39128d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:47:07.509 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.519+0000 7fe39128d640 1 -- 172.21.15.57:0/900243343 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe38c1548e0 con 0x7fe38c0b1f60 2024-03-29T10:47:07.509 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.519+0000 7fe39128d640 1 librados: init done 2024-03-29T10:47:07.509 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.521+0000 7fe39128d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:07.509 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.521+0000 7fe39128d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:07.509 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.521+0000 7fe39128d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:47:07.509 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.521+0000 7fe39128d640 1 -- 172.21.15.57:0/900243343 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe38c149840 con 0x7fe38c0b1f60 2024-03-29T10:47:07.509 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.522+0000 7fe3737fe640 1 -- 172.21.15.57:0/900243343 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe384004e80 con 0x7fe38c0b1f60 2024-03-29T10:47:07.509 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.522+0000 7fe3737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:07.509 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.522+0000 7fe3737fe640 1 -- 172.21.15.57:0/900243343 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fe384005130 con 0x7fe38c0b1f60 2024-03-29T10:47:07.509 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.522+0000 7fe3737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:07.510 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.522+0000 7fe3737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:07.510 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.522+0000 7fe3737fe640 10 monclient: dump: 2024-03-29T10:47:07.510 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:07.510 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:07.510 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:07.510 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:07.510 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:07.510 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:07.510 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:07.510 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:07.510 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:07.510 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:07.510 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.522+0000 7fe3897fa640 4 set_mon_vals no callback set 2024-03-29T10:47:07.511 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.525+0000 7fe3737fe640 1 -- 172.21.15.57:0/900243343 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fe384005290 con 0x7fe38c0b1f60 2024-03-29T10:47:07.511 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.526+0000 7fe3737fe640 1 -- 172.21.15.57:0/900243343 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7fe384084dc0 con 0x7fe38c0b1f60 2024-03-29T10:47:07.511 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.526+0000 7fe3737fe640 10 client.4520.objecter ms_dispatch 0x7fe38c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:07.511 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.526+0000 7fe38a7fc640 10 monclient: get_auth_request con 0x7fe37806ddc0 auth_method 0 2024-03-29T10:47:07.511 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.526+0000 7fe3737fe640 3 client.4520.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:07.511 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.526+0000 7fe3737fe640 3 client.4520.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:07.511 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.526+0000 7fe3737fe640 20 client.4520.objecter dump_active .. 0 homeless 2024-03-29T10:47:07.512 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.526+0000 7fe3737fe640 10 client.4520.objecter ms_handle_connect 0x7fe37806ddc0 2024-03-29T10:47:07.515 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.533+0000 7fe3737fe640 1 -- 172.21.15.57:0/900243343 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe3840506a0 con 0x7fe38c0b1f60 2024-03-29T10:47:07.515 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.533+0000 7fe3737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:07.515 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.533+0000 7fe3737fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:07.844 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.861+0000 7fe39128d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:07.844 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.861+0000 7fe39128d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:07.844 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.861+0000 7fe39128d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:47:07.844 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.861+0000 7fe39128d640 1 -- 172.21.15.57:0/900243343 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe38c075f10 con 0x7fe38c0b1f60 2024-03-29T10:47:07.845 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.862+0000 7fe3737fe640 1 -- 172.21.15.57:0/900243343 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7fe3840525d0 con 0x7fe38c0b1f60 2024-03-29T10:47:07.845 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.863+0000 7fe3737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:07.845 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.863+0000 7fe3737fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:07.845 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:47:07.846 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:07.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.866+0000 7fe39128d640 10 librados: watch_flush enter 2024-03-29T10:47:07.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.866+0000 7fe39128d640 10 librados: watch_flush exit 2024-03-29T10:47:07.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.866+0000 7fe39128d640 20 client.4520.objecter shutdown clearing up homeless session... 2024-03-29T10:47:07.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.866+0000 7fe39128d640 10 client.4520.objecter successfully canceled tick 2024-03-29T10:47:07.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.866+0000 7fe39128d640 1 -- 172.21.15.57:0/900243343 >> v1:172.21.15.179:6816/2698295936 conn(0x7fe37806ddc0 legacy=0x7fe378070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:07.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.866+0000 7fe39128d640 10 monclient: shutdown 2024-03-29T10:47:07.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.866+0000 7fe39128d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:07.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.866+0000 7fe39128d640 1 -- 172.21.15.57:0/900243343 >> v1:172.21.15.57:6790/0 conn(0x7fe38c0b1f60 legacy=0x7fe38c13f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:07.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.866+0000 7fe39128d640 1 -- 172.21.15.57:0/900243343 shutdown_connections 2024-03-29T10:47:07.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.866+0000 7fe39128d640 1 -- 172.21.15.57:0/900243343 >> 172.21.15.57:0/900243343 conn(0x7fe38c066a00 msgr2=0x7fe38c146a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:07.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.867+0000 7fe39128d640 1 -- 172.21.15.57:0/900243343 shutdown_connections 2024-03-29T10:47:07.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.867+0000 7fe39128d640 1 -- 172.21.15.57:0/900243343 wait complete. 2024-03-29T10:47:07.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:07.867+0000 7fe39128d640 1 librados: shutdown 2024-03-29T10:47:07.870 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:14.677 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:14.829 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.846+0000 7fd3712fb640 10 monclient: get_monmap_and_config 2024-03-29T10:47:14.829 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.846+0000 7fd3712fb640 10 monclient: build_initial_monmap 2024-03-29T10:47:14.829 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.846+0000 7fd3712fb640 10 monclient: monmap: 2024-03-29T10:47:14.829 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:47:14.829 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:14.829 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:47:14.847396+0000 2024-03-29T10:47:14.829 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:47:14.847396+0000 2024-03-29T10:47:14.829 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:14.829 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:14.830 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:47:14.830 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:47:14.830 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:47:14.830 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:14.830 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.847+0000 7fd3712fb640 1 Processor -- start 2024-03-29T10:47:14.830 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.848+0000 7fd3712fb640 1 -- start start 2024-03-29T10:47:14.831 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.848+0000 7fd3712fb640 10 monclient: init 2024-03-29T10:47:14.832 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.849+0000 7fd3712fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:14.833 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.849+0000 7fd3712fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:14.833 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.849+0000 7fd3712fb640 10 monclient(hunting): picked mon.noname-a con 0x7fd36c1457f0 addr v1:172.21.15.57:6789/0 2024-03-29T10:47:14.833 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.849+0000 7fd3712fb640 10 monclient(hunting): picked mon.noname-c con 0x7fd36c13f650 addr v1:172.21.15.179:6789/0 2024-03-29T10:47:14.833 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.849+0000 7fd3712fb640 10 monclient(hunting): picked mon.noname-b con 0x7fd36c0a9d70 addr v1:172.21.15.57:6790/0 2024-03-29T10:47:14.833 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.849+0000 7fd3712fb640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd36c0aaef0 con 0x7fd36c1457f0 2024-03-29T10:47:14.833 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.849+0000 7fd3712fb640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd36c145bd0 con 0x7fd36c13f650 2024-03-29T10:47:14.833 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.849+0000 7fd3712fb640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd36c145eb0 con 0x7fd36c0a9d70 2024-03-29T10:47:14.833 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.849+0000 7fd3712fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:14.833 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.849+0000 7fd3712fb640 10 monclient(hunting): authenticate will time out at 1387.025474s 2024-03-29T10:47:14.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.849+0000 7fd36a7fc640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fd36c13f650 0x7fd36c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:40682/0 (socket says 172.21.15.57:40682) 2024-03-29T10:47:14.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.849+0000 7fd36a7fc640 1 -- 172.21.15.57:0/2411415326 learned_addr learned my addr 172.21.15.57:0/2411415326 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:47:14.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.850+0000 7fd369ffb640 1 -- 172.21.15.57:0/2411415326 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462212480 0 0) 0x7fd36c145bd0 con 0x7fd36c13f650 2024-03-29T10:47:14.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.850+0000 7fd369ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:14.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.850+0000 7fd369ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:14.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.850+0000 7fd369ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:14.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.850+0000 7fd369ffb640 10 monclient(hunting): my global_id is 4489 2024-03-29T10:47:14.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.850+0000 7fd369ffb640 1 -- 172.21.15.57:0/2411415326 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd358003520 con 0x7fd36c13f650 2024-03-29T10:47:14.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.850+0000 7fd369ffb640 1 -- 172.21.15.57:0/2411415326 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128119029 0 0) 0x7fd36c0aaef0 con 0x7fd36c1457f0 2024-03-29T10:47:14.834 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.850+0000 7fd369ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:14.835 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.850+0000 7fd369ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:14.835 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.850+0000 7fd369ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:14.835 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.850+0000 7fd369ffb640 10 monclient(hunting): my global_id is 4479 2024-03-29T10:47:14.835 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.850+0000 7fd369ffb640 1 -- 172.21.15.57:0/2411415326 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd36c145bd0 con 0x7fd36c1457f0 2024-03-29T10:47:14.836 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.851+0000 7fd369ffb640 1 -- 172.21.15.57:0/2411415326 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4002544977 0 0) 0x7fd358003520 con 0x7fd36c13f650 2024-03-29T10:47:14.836 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.851+0000 7fd369ffb640 1 -- 172.21.15.57:0/2411415326 >> v1:172.21.15.57:6790/0 conn(0x7fd36c0a9d70 legacy=0x7fd36c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:14.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.851+0000 7fd369ffb640 1 -- 172.21.15.57:0/2411415326 >> v1:172.21.15.57:6789/0 conn(0x7fd36c1457f0 legacy=0x7fd36c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:14.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.851+0000 7fd369ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:14.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.851+0000 7fd369ffb640 1 monclient: found mon.noname-c 2024-03-29T10:47:14.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.851+0000 7fd369ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:14.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.851+0000 7fd369ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:47:14.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.851+0000 7fd369ffb640 1 -- 172.21.15.57:0/2411415326 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd36c146340 con 0x7fd36c13f650 2024-03-29T10:47:14.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.851+0000 7fd369ffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:14.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.851+0000 7fd369ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:14.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.851+0000 7fd369ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:14.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.851+0000 7fd369ffb640 1 -- 172.21.15.57:0/2411415326 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd354001690 con 0x7fd36c13f650 2024-03-29T10:47:14.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.851+0000 7fd369ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:14.837 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.851+0000 7fd369ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:14.838 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.851+0000 7fd369ffb640 10 monclient: dump: 2024-03-29T10:47:14.838 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:14.838 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:14.838 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:14.838 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:14.838 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:14.838 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:14.838 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:14.838 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:14.838 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:14.839 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:14.839 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.851+0000 7fd3712fb640 5 monclient: authenticate success, global_id 4489 2024-03-29T10:47:14.839 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.851+0000 7fd3712fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:14.839 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.852+0000 7fd369ffb640 1 -- 172.21.15.57:0/2411415326 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd354001870 con 0x7fd36c13f650 2024-03-29T10:47:14.839 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.852+0000 7fd369ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:14.839 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.852+0000 7fd369ffb640 1 -- 172.21.15.57:0/2411415326 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd354001cc0 con 0x7fd36c13f650 2024-03-29T10:47:14.839 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.852+0000 7fd369ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:14.839 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.852+0000 7fd369ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:14.839 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.852+0000 7fd369ffb640 10 monclient: dump: 2024-03-29T10:47:14.839 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:14.839 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:14.840 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:14.840 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:14.840 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:14.840 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:14.840 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:14.840 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:14.840 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:14.840 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:14.840 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.852+0000 7fd3712fb640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:14.840 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.852+0000 7fd3712fb640 4 set_mon_vals no callback set 2024-03-29T10:47:14.840 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.852+0000 7fd3712fb640 10 monclient: shutdown 2024-03-29T10:47:14.840 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.852+0000 7fd3712fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:14.840 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.852+0000 7fd3712fb640 1 -- 172.21.15.57:0/2411415326 >> v1:172.21.15.179:6789/0 conn(0x7fd36c13f650 legacy=0x7fd36c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:14.841 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.853+0000 7fd3712fb640 1 -- 172.21.15.57:0/2411415326 shutdown_connections 2024-03-29T10:47:14.841 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.853+0000 7fd3712fb640 1 -- 172.21.15.57:0/2411415326 >> 172.21.15.57:0/2411415326 conn(0x7fd36c066a00 msgr2=0x7fd36c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:14.841 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.853+0000 7fd3712fb640 1 -- 172.21.15.57:0/2411415326 shutdown_connections 2024-03-29T10:47:14.841 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.853+0000 7fd3712fb640 1 -- 172.21.15.57:0/2411415326 wait complete. 2024-03-29T10:47:14.841 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.854+0000 7fd3712fb640 10 monclient: build_initial_monmap 2024-03-29T10:47:14.841 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.854+0000 7fd3712fb640 10 monclient: monmap: 2024-03-29T10:47:14.841 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:47:14.841 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:14.841 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:47:14.841 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:47:14.841 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:14.841 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:14.841 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:47:14.841 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:47:14.841 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:47:14.842 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:14.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.854+0000 7fd3712fb640 1 librados: starting msgr at 2024-03-29T10:47:14.842 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.854+0000 7fd3712fb640 1 librados: starting objecter 2024-03-29T10:47:14.848 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.855+0000 7fd3712fb640 1 Processor -- start 2024-03-29T10:47:14.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.855+0000 7fd3712fb640 1 -- start start 2024-03-29T10:47:14.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.855+0000 7fd3712fb640 1 librados: setting wanted keys 2024-03-29T10:47:14.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.855+0000 7fd3712fb640 1 librados: calling monclient init 2024-03-29T10:47:14.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.855+0000 7fd3712fb640 10 monclient: init 2024-03-29T10:47:14.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.856+0000 7fd3712fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:14.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.856+0000 7fd3712fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:14.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.856+0000 7fd3712fb640 10 monclient(hunting): picked mon.noname-a con 0x7fd36c0a9d70 addr v1:172.21.15.57:6789/0 2024-03-29T10:47:14.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.856+0000 7fd3712fb640 10 monclient(hunting): picked mon.noname-c con 0x7fd36c1457f0 addr v1:172.21.15.57:6790/0 2024-03-29T10:47:14.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.856+0000 7fd3712fb640 10 monclient(hunting): picked mon.noname-b con 0x7fd36c1dc4e0 addr v1:172.21.15.179:6789/0 2024-03-29T10:47:14.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.856+0000 7fd3712fb640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd36c06de90 con 0x7fd36c0a9d70 2024-03-29T10:47:14.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.856+0000 7fd3712fb640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd36c070190 con 0x7fd36c1dc4e0 2024-03-29T10:47:14.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.856+0000 7fd3712fb640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd36c0b1770 con 0x7fd36c1457f0 2024-03-29T10:47:14.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.856+0000 7fd3712fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:14.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.856+0000 7fd3712fb640 10 monclient(hunting): authenticate will time out at 1387.032396s 2024-03-29T10:47:14.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.856+0000 7fd36b7fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fd36c1dc4e0 0x7fd36c1e6430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:41124/0 (socket says 172.21.15.57:41124) 2024-03-29T10:47:14.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.856+0000 7fd36b7fe640 1 -- 172.21.15.57:0/1653125570 learned_addr learned my addr 172.21.15.57:0/1653125570 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:47:14.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.856+0000 7fd3637fe640 10 client.?.objecter ms_handle_connect 0x7fd36c1dc4e0 2024-03-29T10:47:14.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.856+0000 7fd3637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:14.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.856+0000 7fd3637fe640 10 client.?.objecter ms_handle_connect 0x7fd36c1457f0 2024-03-29T10:47:14.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.856+0000 7fd3637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:14.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.856+0000 7fd3637fe640 10 client.?.objecter ms_handle_connect 0x7fd36c0a9d70 2024-03-29T10:47:14.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.856+0000 7fd3637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:14.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.857+0000 7fd3637fe640 1 -- 172.21.15.57:0/1653125570 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 413166674 0 0) 0x7fd36c070190 con 0x7fd36c1dc4e0 2024-03-29T10:47:14.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.857+0000 7fd3637fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:14.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.857+0000 7fd3637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:14.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.857+0000 7fd3637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:14.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.857+0000 7fd3637fe640 10 monclient(hunting): my global_id is 4492 2024-03-29T10:47:14.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.857+0000 7fd3637fe640 1 -- 172.21.15.57:0/1653125570 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd34c004eb0 con 0x7fd36c1dc4e0 2024-03-29T10:47:14.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.857+0000 7fd3637fe640 1 -- 172.21.15.57:0/1653125570 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2319308872 0 0) 0x7fd36c0b1770 con 0x7fd36c1457f0 2024-03-29T10:47:14.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.857+0000 7fd3637fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:14.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.857+0000 7fd3637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:14.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.857+0000 7fd3637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:14.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.857+0000 7fd3637fe640 10 monclient(hunting): my global_id is 4526 2024-03-29T10:47:14.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.857+0000 7fd3637fe640 1 -- 172.21.15.57:0/1653125570 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd36c070190 con 0x7fd36c1457f0 2024-03-29T10:47:14.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.857+0000 7fd3637fe640 1 -- 172.21.15.57:0/1653125570 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 797419901 0 0) 0x7fd34c004eb0 con 0x7fd36c1dc4e0 2024-03-29T10:47:14.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.857+0000 7fd3637fe640 1 -- 172.21.15.57:0/1653125570 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd36c0b1770 con 0x7fd36c1dc4e0 2024-03-29T10:47:14.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.857+0000 7fd3637fe640 1 -- 172.21.15.57:0/1653125570 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd364001c90 con 0x7fd36c1dc4e0 2024-03-29T10:47:14.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.857+0000 7fd3637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:14.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.857+0000 7fd3637fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:14.852 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.857+0000 7fd3637fe640 10 monclient(hunting): dump: 2024-03-29T10:47:14.852 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:14.852 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:14.852 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:14.852 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:14.852 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:14.852 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:14.852 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:14.852 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:14.852 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:14.853 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:14.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.857+0000 7fd3637fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:14.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.857+0000 7fd3637fe640 1 -- 172.21.15.57:0/1653125570 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 248878379 0 0) 0x7fd36c070190 con 0x7fd36c1457f0 2024-03-29T10:47:14.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.857+0000 7fd3637fe640 1 -- 172.21.15.57:0/1653125570 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd34c004eb0 con 0x7fd36c1457f0 2024-03-29T10:47:14.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3637fe640 1 -- 172.21.15.57:0/1653125570 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd354001b80 con 0x7fd36c1457f0 2024-03-29T10:47:14.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:14.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3637fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:14.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3637fe640 10 monclient(hunting): dump: 2024-03-29T10:47:14.853 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:14.853 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:14.853 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:14.853 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:14.853 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:14.853 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:14.854 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:14.854 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:14.854 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:14.854 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:14.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3637fe640 1 -- 172.21.15.57:0/1653125570 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276552399 0 0) 0x7fd36c06de90 con 0x7fd36c0a9d70 2024-03-29T10:47:14.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3637fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:14.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:14.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:14.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3637fe640 10 monclient(hunting): my global_id is 4482 2024-03-29T10:47:14.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3637fe640 1 -- 172.21.15.57:0/1653125570 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd36c070190 con 0x7fd36c0a9d70 2024-03-29T10:47:14.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3637fe640 1 -- 172.21.15.57:0/1653125570 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2488309037 0 0) 0x7fd36c0b1770 con 0x7fd36c1dc4e0 2024-03-29T10:47:14.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3637fe640 1 -- 172.21.15.57:0/1653125570 >> v1:172.21.15.57:6790/0 conn(0x7fd36c1457f0 legacy=0x7fd36c1dbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:14.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3637fe640 1 -- 172.21.15.57:0/1653125570 >> v1:172.21.15.57:6789/0 conn(0x7fd36c0a9d70 legacy=0x7fd36c070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:14.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3637fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:14.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3637fe640 1 monclient: found mon.b 2024-03-29T10:47:14.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:14.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3637fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:47:14.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3637fe640 1 -- 172.21.15.57:0/1653125570 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd36c0712f0 con 0x7fd36c1dc4e0 2024-03-29T10:47:14.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3637fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:14.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3637fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:14.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:14.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3712fb640 5 monclient: authenticate success, global_id 4492 2024-03-29T10:47:14.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3712fb640 10 monclient: _renew_subs 2024-03-29T10:47:14.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3712fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:47:14.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3712fb640 1 -- 172.21.15.57:0/1653125570 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd36c1e7b50 con 0x7fd36c1dc4e0 2024-03-29T10:47:14.855 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3712fb640 10 client.4492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:14.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3712fb640 10 monclient: _renew_subs 2024-03-29T10:47:14.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3712fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:47:14.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3712fb640 1 -- 172.21.15.57:0/1653125570 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd36c1e7ec0 con 0x7fd36c1dc4e0 2024-03-29T10:47:14.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3712fb640 1 librados: init done 2024-03-29T10:47:14.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3637fe640 1 -- 172.21.15.57:0/1653125570 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd364003990 con 0x7fd36c1dc4e0 2024-03-29T10:47:14.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:14.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.858+0000 7fd3697fa640 4 set_mon_vals no callback set 2024-03-29T10:47:14.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.859+0000 7fd3637fe640 1 -- 172.21.15.57:0/1653125570 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd364003da0 con 0x7fd36c1dc4e0 2024-03-29T10:47:14.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.859+0000 7fd3637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:14.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.859+0000 7fd3637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:47:14.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.859+0000 7fd3637fe640 10 monclient: dump: 2024-03-29T10:47:14.857 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:14.857 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:14.857 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:14.857 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:14.857 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:14.857 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:14.857 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:14.857 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:14.857 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:14.857 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:14.857 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.861+0000 7fd3637fe640 1 -- 172.21.15.57:0/1653125570 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fd364003fc0 con 0x7fd36c1dc4e0 2024-03-29T10:47:14.857 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.861+0000 7fd3637fe640 1 -- 172.21.15.57:0/1653125570 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7fd364084d60 con 0x7fd36c1dc4e0 2024-03-29T10:47:14.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.861+0000 7fd3637fe640 10 client.4492.objecter ms_dispatch 0x7fd36c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:14.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.861+0000 7fd3637fe640 3 client.4492.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:14.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.861+0000 7fd3637fe640 3 client.4492.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:14.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.861+0000 7fd3637fe640 20 client.4492.objecter dump_active .. 0 homeless 2024-03-29T10:47:14.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.861+0000 7fd3712fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:14.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.861+0000 7fd3712fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:14.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.861+0000 7fd3712fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:47:14.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.861+0000 7fd3712fb640 1 -- 172.21.15.57:0/1653125570 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd350005080 con 0x7fd36c1dc4e0 2024-03-29T10:47:14.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.862+0000 7fd36affd640 10 monclient: get_auth_request con 0x7fd34c06e130 auth_method 0 2024-03-29T10:47:14.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.862+0000 7fd3637fe640 10 client.4492.objecter ms_handle_connect 0x7fd34c06e130 2024-03-29T10:47:14.859 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.869+0000 7fd3637fe640 1 -- 172.21.15.57:0/1653125570 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd364050640 con 0x7fd36c1dc4e0 2024-03-29T10:47:14.859 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.869+0000 7fd3637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:14.859 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:14.869+0000 7fd3637fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:15.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:15.249+0000 7fd3712fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:15.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:15.249+0000 7fd3712fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:15.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:15.249+0000 7fd3712fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:47:15.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:15.249+0000 7fd3712fb640 1 -- 172.21.15.57:0/1653125570 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd350005330 con 0x7fd36c1dc4e0 2024-03-29T10:47:15.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:15.250+0000 7fd3637fe640 1 -- 172.21.15.57:0/1653125570 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7fd364052570 con 0x7fd36c1dc4e0 2024-03-29T10:47:15.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:15.250+0000 7fd3637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:15.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:15.250+0000 7fd3637fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:15.233 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:47:15.233 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:15.236 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:15.253+0000 7fd3712fb640 10 librados: watch_flush enter 2024-03-29T10:47:15.236 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:15.253+0000 7fd3712fb640 10 librados: watch_flush exit 2024-03-29T10:47:15.236 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:15.253+0000 7fd3712fb640 20 client.4492.objecter shutdown clearing up homeless session... 2024-03-29T10:47:15.236 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:15.253+0000 7fd3712fb640 10 client.4492.objecter successfully canceled tick 2024-03-29T10:47:15.236 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:15.254+0000 7fd3712fb640 1 -- 172.21.15.57:0/1653125570 >> v1:172.21.15.179:6816/2698295936 conn(0x7fd34c06e130 legacy=0x7fd34c0705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:15.237 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:15.254+0000 7fd3712fb640 10 monclient: shutdown 2024-03-29T10:47:15.237 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:15.254+0000 7fd3712fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:15.237 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:15.254+0000 7fd3712fb640 1 -- 172.21.15.57:0/1653125570 >> v1:172.21.15.179:6789/0 conn(0x7fd36c1dc4e0 legacy=0x7fd36c1e6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:15.237 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:15.254+0000 7fd3712fb640 1 -- 172.21.15.57:0/1653125570 shutdown_connections 2024-03-29T10:47:15.237 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:15.254+0000 7fd3712fb640 1 -- 172.21.15.57:0/1653125570 >> 172.21.15.57:0/1653125570 conn(0x7fd36c066a00 msgr2=0x7fd36c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:15.237 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:15.254+0000 7fd3712fb640 1 -- 172.21.15.57:0/1653125570 shutdown_connections 2024-03-29T10:47:15.237 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:15.254+0000 7fd3712fb640 1 -- 172.21.15.57:0/1653125570 wait complete. 2024-03-29T10:47:15.237 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:15.254+0000 7fd3712fb640 1 librados: shutdown 2024-03-29T10:47:15.255 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:22.062 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:22.195 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.213+0000 7fef227ac640 10 monclient: get_monmap_and_config 2024-03-29T10:47:22.195 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.213+0000 7fef227ac640 10 monclient: build_initial_monmap 2024-03-29T10:47:22.195 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.213+0000 7fef227ac640 10 monclient: monmap: 2024-03-29T10:47:22.196 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:47:22.196 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:22.196 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:47:22.214145+0000 2024-03-29T10:47:22.196 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:47:22.214145+0000 2024-03-29T10:47:22.196 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:22.196 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:22.196 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:47:22.196 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:47:22.196 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:47:22.196 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:22.197 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.214+0000 7fef227ac640 1 Processor -- start 2024-03-29T10:47:22.197 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.214+0000 7fef227ac640 1 -- start start 2024-03-29T10:47:22.197 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.214+0000 7fef227ac640 10 monclient: init 2024-03-29T10:47:22.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.215+0000 7fef227ac640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:22.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.215+0000 7fef227ac640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:47:22.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.215+0000 7fef227ac640 10 monclient(hunting): picked mon.noname-c con 0x7fef1c06cac0 addr v1:172.21.15.179:6789/0 2024-03-29T10:47:22.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.215+0000 7fef227ac640 10 monclient(hunting): picked mon.noname-b con 0x7fef1c06cea0 addr v1:172.21.15.57:6790/0 2024-03-29T10:47:22.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.215+0000 7fef227ac640 10 monclient(hunting): picked mon.noname-a con 0x7fef1c145e80 addr v1:172.21.15.57:6789/0 2024-03-29T10:47:22.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.215+0000 7fef227ac640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef1c0ac850 con 0x7fef1c145e80 2024-03-29T10:47:22.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.215+0000 7fef227ac640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef1c070470 con 0x7fef1c06cac0 2024-03-29T10:47:22.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.215+0000 7fef227ac640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef1c072600 con 0x7fef1c06cea0 2024-03-29T10:47:22.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.215+0000 7fef227ac640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:22.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.215+0000 7fef1b7fe640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7fef1c06cea0 0x7fef1c145770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:36474/0 (socket says 172.21.15.57:36474) 2024-03-29T10:47:22.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.215+0000 7fef1b7fe640 1 -- 172.21.15.57:0/1800576180 learned_addr learned my addr 172.21.15.57:0/1800576180 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:47:22.198 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.215+0000 7fef227ac640 10 monclient(hunting): authenticate will time out at 1394.391701s 2024-03-29T10:47:22.200 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.216+0000 7fef1affd640 1 -- 172.21.15.57:0/1800576180 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688629354 0 0) 0x7fef1c072600 con 0x7fef1c06cea0 2024-03-29T10:47:22.200 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.216+0000 7fef1affd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:22.200 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.216+0000 7fef1affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:22.200 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.216+0000 7fef1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:22.200 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.216+0000 7fef1affd640 10 monclient(hunting): my global_id is 4529 2024-03-29T10:47:22.200 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.216+0000 7fef1affd640 1 -- 172.21.15.57:0/1800576180 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef04003520 con 0x7fef1c06cea0 2024-03-29T10:47:22.200 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.216+0000 7fef1affd640 1 -- 172.21.15.57:0/1800576180 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2953522624 0 0) 0x7fef1c0ac850 con 0x7fef1c145e80 2024-03-29T10:47:22.201 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.216+0000 7fef1affd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:22.201 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.216+0000 7fef1affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:22.201 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.216+0000 7fef1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:22.201 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.216+0000 7fef1affd640 10 monclient(hunting): my global_id is 4485 2024-03-29T10:47:22.201 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.216+0000 7fef1affd640 1 -- 172.21.15.57:0/1800576180 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef1c072600 con 0x7fef1c145e80 2024-03-29T10:47:22.201 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.217+0000 7fef1affd640 1 -- 172.21.15.57:0/1800576180 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1862897661 0 0) 0x7fef04003520 con 0x7fef1c06cea0 2024-03-29T10:47:22.201 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.217+0000 7fef1affd640 1 -- 172.21.15.57:0/1800576180 >> v1:172.21.15.179:6789/0 conn(0x7fef1c06cac0 legacy=0x7fef1c071ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:22.203 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.219+0000 7fef1affd640 1 -- 172.21.15.57:0/1800576180 >> v1:172.21.15.57:6789/0 conn(0x7fef1c145e80 legacy=0x7fef1c14db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:22.203 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.219+0000 7fef1affd640 10 monclient: _finish_hunting 0 2024-03-29T10:47:22.203 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.219+0000 7fef1affd640 1 monclient: found mon.noname-b 2024-03-29T10:47:22.203 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.219+0000 7fef1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:22.203 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.219+0000 7fef1affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:47:22.203 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.219+0000 7fef1affd640 1 -- 172.21.15.57:0/1800576180 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef1c1462d0 con 0x7fef1c06cea0 2024-03-29T10:47:22.204 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.219+0000 7fef1affd640 10 monclient: _finish_auth 0 2024-03-29T10:47:22.204 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.219+0000 7fef1affd640 10 monclient: _check_auth_tickets 2024-03-29T10:47:22.204 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.219+0000 7fef1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:22.204 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.219+0000 7fef1affd640 1 -- 172.21.15.57:0/1800576180 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fef0c001690 con 0x7fef1c06cea0 2024-03-29T10:47:22.204 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.219+0000 7fef1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:22.204 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.219+0000 7fef1affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:22.204 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.219+0000 7fef1affd640 10 monclient: dump: 2024-03-29T10:47:22.204 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:22.204 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:22.204 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:22.204 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:22.204 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:22.204 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:22.204 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:22.205 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:22.205 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:22.205 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:22.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.219+0000 7fef227ac640 5 monclient: authenticate success, global_id 4529 2024-03-29T10:47:22.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.219+0000 7fef227ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:22.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.219+0000 7fef1affd640 1 -- 172.21.15.57:0/1800576180 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef0c0017f0 con 0x7fef1c06cea0 2024-03-29T10:47:22.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.219+0000 7fef1affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:22.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.219+0000 7fef227ac640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:22.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.219+0000 7fef1affd640 1 -- 172.21.15.57:0/1800576180 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fef0c001c40 con 0x7fef1c06cea0 2024-03-29T10:47:22.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.219+0000 7fef1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:22.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.219+0000 7fef227ac640 4 set_mon_vals no callback set 2024-03-29T10:47:22.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.219+0000 7fef1affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:22.205 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.219+0000 7fef1affd640 10 monclient: dump: 2024-03-29T10:47:22.205 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:22.206 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:22.206 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:22.206 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:22.206 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:22.206 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:22.206 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:22.206 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:22.206 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:22.206 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:22.206 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.220+0000 7fef227ac640 10 monclient: shutdown 2024-03-29T10:47:22.206 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.220+0000 7fef227ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:22.206 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.220+0000 7fef227ac640 1 -- 172.21.15.57:0/1800576180 >> v1:172.21.15.57:6790/0 conn(0x7fef1c06cea0 legacy=0x7fef1c145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:22.206 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.220+0000 7fef227ac640 1 -- 172.21.15.57:0/1800576180 shutdown_connections 2024-03-29T10:47:22.206 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.220+0000 7fef227ac640 1 -- 172.21.15.57:0/1800576180 >> 172.21.15.57:0/1800576180 conn(0x7fef1c0675b0 msgr2=0x7fef1c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:22.206 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.220+0000 7fef227ac640 1 -- 172.21.15.57:0/1800576180 shutdown_connections 2024-03-29T10:47:22.207 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.221+0000 7fef227ac640 1 -- 172.21.15.57:0/1800576180 wait complete. 2024-03-29T10:47:22.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.221+0000 7fef227ac640 10 monclient: build_initial_monmap 2024-03-29T10:47:22.209 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.221+0000 7fef227ac640 10 monclient: monmap: 2024-03-29T10:47:22.209 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:47:22.210 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:22.210 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:47:22.210 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:47:22.210 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:22.210 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:22.210 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:47:22.210 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:47:22.210 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:47:22.210 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:22.210 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.222+0000 7fef227ac640 1 librados: starting msgr at 2024-03-29T10:47:22.210 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.222+0000 7fef227ac640 1 librados: starting objecter 2024-03-29T10:47:22.210 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.222+0000 7fef227ac640 1 Processor -- start 2024-03-29T10:47:22.210 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.222+0000 7fef227ac640 1 -- start start 2024-03-29T10:47:22.211 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.222+0000 7fef227ac640 1 librados: setting wanted keys 2024-03-29T10:47:22.211 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.222+0000 7fef227ac640 1 librados: calling monclient init 2024-03-29T10:47:22.211 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.222+0000 7fef227ac640 10 monclient: init 2024-03-29T10:47:22.211 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.223+0000 7fef227ac640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:22.211 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.223+0000 7fef227ac640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:47:22.211 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.223+0000 7fef227ac640 10 monclient(hunting): picked mon.noname-c con 0x7fef1c06cac0 addr v1:172.21.15.57:6790/0 2024-03-29T10:47:22.211 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.223+0000 7fef227ac640 10 monclient(hunting): picked mon.noname-b con 0x7fef1c06cea0 addr v1:172.21.15.179:6789/0 2024-03-29T10:47:22.211 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.223+0000 7fef227ac640 10 monclient(hunting): picked mon.noname-a con 0x7fef1c145e80 addr v1:172.21.15.57:6789/0 2024-03-29T10:47:22.211 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.223+0000 7fef227ac640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef1c0ac1f0 con 0x7fef1c145e80 2024-03-29T10:47:22.211 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.223+0000 7fef227ac640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef1c0ab7b0 con 0x7fef1c06cea0 2024-03-29T10:47:22.211 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.223+0000 7fef227ac640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef1c13bb10 con 0x7fef1c06cac0 2024-03-29T10:47:22.211 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.223+0000 7fef227ac640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:22.211 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.223+0000 7fef227ac640 10 monclient(hunting): authenticate will time out at 1394.399511s 2024-03-29T10:47:22.212 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.224+0000 7fef1bfff640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7fef1c06cac0 0x7fef1c142b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:36878/0 (socket says 172.21.15.57:36878) 2024-03-29T10:47:22.212 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.225+0000 7fef1bfff640 1 -- 172.21.15.57:0/626039401 learned_addr learned my addr 172.21.15.57:0/626039401 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:47:22.212 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.225+0000 7fef18ff9640 10 client.?.objecter ms_handle_connect 0x7fef1c06cac0 2024-03-29T10:47:22.212 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.225+0000 7fef18ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:22.212 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.225+0000 7fef18ff9640 1 -- 172.21.15.57:0/626039401 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 935726117 0 0) 0x7fef1c13bb10 con 0x7fef1c06cac0 2024-03-29T10:47:22.212 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.225+0000 7fef18ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:47:22.212 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.225+0000 7fef18ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:22.212 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.225+0000 7fef18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:22.212 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.225+0000 7fef18ff9640 10 monclient(hunting): my global_id is 4532 2024-03-29T10:47:22.212 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.225+0000 7fef18ff9640 1 -- 172.21.15.57:0/626039401 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef08004eb0 con 0x7fef1c06cac0 2024-03-29T10:47:22.212 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.225+0000 7fef18ff9640 1 -- 172.21.15.57:0/626039401 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4289708838 0 0) 0x7fef08004eb0 con 0x7fef1c06cac0 2024-03-29T10:47:22.212 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.226+0000 7fef18ff9640 1 -- 172.21.15.57:0/626039401 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef1c13bb10 con 0x7fef1c06cac0 2024-03-29T10:47:22.212 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.226+0000 7fef18ff9640 1 -- 172.21.15.57:0/626039401 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fef14001bf0 con 0x7fef1c06cac0 2024-03-29T10:47:22.212 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.226+0000 7fef18ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:22.212 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.226+0000 7fef18ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:22.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.226+0000 7fef18ff9640 10 monclient(hunting): dump: 2024-03-29T10:47:22.213 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:22.213 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:22.213 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:22.213 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:22.213 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:22.213 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:22.213 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:22.213 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:22.213 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:22.213 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:22.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.226+0000 7fef18ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:22.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.226+0000 7fef18ff9640 1 -- 172.21.15.57:0/626039401 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2925304947 0 0) 0x7fef1c13bb10 con 0x7fef1c06cac0 2024-03-29T10:47:22.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.226+0000 7fef18ff9640 1 -- 172.21.15.57:0/626039401 >> v1:172.21.15.179:6789/0 conn(0x7fef1c06cea0 legacy=0x7fef1c0aae90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:47:22.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef18ff9640 1 -- 172.21.15.57:0/626039401 >> v1:172.21.15.57:6789/0 conn(0x7fef1c145e80 legacy=0x7fef1c13b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:22.214 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef18ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:47:22.214 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef18ff9640 1 monclient: found mon.c 2024-03-29T10:47:22.214 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef18ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:22.214 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef18ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:47:22.214 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef18ff9640 1 -- 172.21.15.57:0/626039401 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef1c0ac420 con 0x7fef1c06cac0 2024-03-29T10:47:22.214 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef18ff9640 10 monclient: _finish_auth 0 2024-03-29T10:47:22.214 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef18ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:47:22.214 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:22.214 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef18ff9640 10 client.?.objecter ms_handle_connect 0x7fef1c145e80 2024-03-29T10:47:22.214 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef18ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:22.214 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef227ac640 5 monclient: authenticate success, global_id 4532 2024-03-29T10:47:22.215 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef227ac640 10 monclient: _renew_subs 2024-03-29T10:47:22.215 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef227ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:47:22.215 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef227ac640 1 -- 172.21.15.57:0/626039401 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef1c13afe0 con 0x7fef1c06cac0 2024-03-29T10:47:22.215 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef227ac640 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:22.215 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef227ac640 10 monclient: _renew_subs 2024-03-29T10:47:22.215 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef227ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:47:22.215 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef227ac640 1 -- 172.21.15.57:0/626039401 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef1c1e3390 con 0x7fef1c06cac0 2024-03-29T10:47:22.215 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef227ac640 1 librados: init done 2024-03-29T10:47:22.215 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef18ff9640 1 -- 172.21.15.57:0/626039401 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef14001d50 con 0x7fef1c06cac0 2024-03-29T10:47:22.215 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef18ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:22.215 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef18ff9640 1 -- 172.21.15.57:0/626039401 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fef14004e60 con 0x7fef1c06cac0 2024-03-29T10:47:22.215 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef18ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:22.215 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef18ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:22.215 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef1a7fc640 4 set_mon_vals no callback set 2024-03-29T10:47:22.215 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.227+0000 7fef18ff9640 10 monclient: dump: 2024-03-29T10:47:22.216 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:22.216 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:22.216 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:22.216 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:22.216 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:22.216 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:22.216 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:22.216 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:22.216 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:22.216 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:22.217 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.229+0000 7fef227ac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:22.217 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.229+0000 7fef227ac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:22.217 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.229+0000 7fef227ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:47:22.217 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.229+0000 7fef227ac640 1 -- 172.21.15.57:0/626039401 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feefc005080 con 0x7fef1c06cac0 2024-03-29T10:47:22.217 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.230+0000 7fef18ff9640 1 -- 172.21.15.57:0/626039401 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fef1401ae30 con 0x7fef1c06cac0 2024-03-29T10:47:22.217 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.231+0000 7fef18ff9640 1 -- 172.21.15.57:0/626039401 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7fef14084e40 con 0x7fef1c06cac0 2024-03-29T10:47:22.217 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.231+0000 7fef18ff9640 10 client.4532.objecter ms_dispatch 0x7fef1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:22.217 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.231+0000 7fef18ff9640 3 client.4532.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:22.218 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.231+0000 7fef18ff9640 3 client.4532.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:22.218 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.231+0000 7fef1b7fe640 10 monclient: get_auth_request con 0x7fef0806db30 auth_method 0 2024-03-29T10:47:22.218 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.231+0000 7fef18ff9640 20 client.4532.objecter dump_active .. 0 homeless 2024-03-29T10:47:22.218 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.232+0000 7fef18ff9640 10 client.4532.objecter ms_handle_connect 0x7fef0806db30 2024-03-29T10:47:22.220 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.238+0000 7fef18ff9640 1 -- 172.21.15.57:0/626039401 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fef14050720 con 0x7fef1c06cac0 2024-03-29T10:47:22.220 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.238+0000 7fef18ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:22.220 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.238+0000 7fef18ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:22.555 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.572+0000 7fef227ac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:22.555 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.572+0000 7fef227ac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:22.555 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.573+0000 7fef227ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:47:22.555 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.573+0000 7fef227ac640 1 -- 172.21.15.57:0/626039401 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feefc005330 con 0x7fef1c06cac0 2024-03-29T10:47:22.556 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.574+0000 7fef18ff9640 1 -- 172.21.15.57:0/626039401 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7fef14052650 con 0x7fef1c06cac0 2024-03-29T10:47:22.556 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.574+0000 7fef18ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:22.556 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.574+0000 7fef18ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:22.556 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:47:22.556 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:22.560 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.577+0000 7fef227ac640 10 librados: watch_flush enter 2024-03-29T10:47:22.560 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.578+0000 7fef227ac640 10 librados: watch_flush exit 2024-03-29T10:47:22.560 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.578+0000 7fef227ac640 20 client.4532.objecter shutdown clearing up homeless session... 2024-03-29T10:47:22.560 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.578+0000 7fef227ac640 10 client.4532.objecter successfully canceled tick 2024-03-29T10:47:22.560 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.578+0000 7fef227ac640 1 -- 172.21.15.57:0/626039401 >> v1:172.21.15.179:6816/2698295936 conn(0x7fef0806db30 legacy=0x7fef0806fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:22.560 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.578+0000 7fef227ac640 10 monclient: shutdown 2024-03-29T10:47:22.560 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.578+0000 7fef227ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:22.560 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.578+0000 7fef227ac640 1 -- 172.21.15.57:0/626039401 >> v1:172.21.15.57:6790/0 conn(0x7fef1c06cac0 legacy=0x7fef1c142b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:22.560 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.578+0000 7fef227ac640 1 -- 172.21.15.57:0/626039401 shutdown_connections 2024-03-29T10:47:22.560 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.578+0000 7fef227ac640 1 -- 172.21.15.57:0/626039401 >> 172.21.15.57:0/626039401 conn(0x7fef1c0675b0 msgr2=0x7fef1c0b0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:22.561 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.578+0000 7fef227ac640 1 -- 172.21.15.57:0/626039401 shutdown_connections 2024-03-29T10:47:22.561 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.578+0000 7fef227ac640 1 -- 172.21.15.57:0/626039401 wait complete. 2024-03-29T10:47:22.561 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:22.578+0000 7fef227ac640 1 librados: shutdown 2024-03-29T10:47:22.576 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:29.384 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:29.528 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.545+0000 7f4aed286640 10 monclient: get_monmap_and_config 2024-03-29T10:47:29.528 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.545+0000 7f4aed286640 10 monclient: build_initial_monmap 2024-03-29T10:47:29.528 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.545+0000 7f4aed286640 10 monclient: monmap: 2024-03-29T10:47:29.528 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:47:29.528 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:29.528 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:47:29.545874+0000 2024-03-29T10:47:29.528 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:47:29.545874+0000 2024-03-29T10:47:29.528 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:29.528 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:29.528 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:47:29.529 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:47:29.529 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:47:29.529 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:29.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.546+0000 7f4aed286640 1 Processor -- start 2024-03-29T10:47:29.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.546+0000 7f4aed286640 1 -- start start 2024-03-29T10:47:29.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.546+0000 7f4aed286640 10 monclient: init 2024-03-29T10:47:29.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.547+0000 7f4aed286640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:29.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.547+0000 7f4aed286640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:47:29.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.547+0000 7f4aed286640 10 monclient(hunting): picked mon.noname-c con 0x7f4ae81457f0 addr v1:172.21.15.179:6789/0 2024-03-29T10:47:29.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.547+0000 7f4aed286640 10 monclient(hunting): picked mon.noname-a con 0x7f4ae813f650 addr v1:172.21.15.57:6789/0 2024-03-29T10:47:29.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.547+0000 7f4aed286640 10 monclient(hunting): picked mon.noname-b con 0x7f4ae80a9d70 addr v1:172.21.15.57:6790/0 2024-03-29T10:47:29.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.547+0000 7f4aed286640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ae80aaef0 con 0x7f4ae813f650 2024-03-29T10:47:29.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.547+0000 7f4aed286640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ae8145bd0 con 0x7f4ae81457f0 2024-03-29T10:47:29.531 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.547+0000 7f4aed286640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ae8145eb0 con 0x7f4ae80a9d70 2024-03-29T10:47:29.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.547+0000 7f4aed286640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:29.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.547+0000 7f4aed286640 10 monclient(hunting): authenticate will time out at 1401.723505s 2024-03-29T10:47:29.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.547+0000 7f4ae6ffd640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f4ae81457f0 0x7f4ae813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:45626/0 (socket says 172.21.15.57:45626) 2024-03-29T10:47:29.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.547+0000 7f4ae6ffd640 1 -- 172.21.15.57:0/3671845110 learned_addr learned my addr 172.21.15.57:0/3671845110 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:47:29.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.547+0000 7f4ae77fe640 1 --1- 172.21.15.57:0/3671845110 >> v1:172.21.15.57:6790/0 conn(0x7f4ae80a9d70 0x7f4ae814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:44100/0 (socket says 172.21.15.57:44100) 2024-03-29T10:47:29.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.548+0000 7f4ae5ffb640 1 -- 172.21.15.57:0/3671845110 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444323531 0 0) 0x7f4ae8145eb0 con 0x7f4ae80a9d70 2024-03-29T10:47:29.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.548+0000 7f4ae5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:29.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.548+0000 7f4ae5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:29.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.548+0000 7f4ae5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:29.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.548+0000 7f4ae5ffb640 10 monclient(hunting): my global_id is 4535 2024-03-29T10:47:29.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.548+0000 7f4ae5ffb640 1 -- 172.21.15.57:0/3671845110 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ad0003520 con 0x7f4ae80a9d70 2024-03-29T10:47:29.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.548+0000 7f4ae5ffb640 1 -- 172.21.15.57:0/3671845110 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1855753083 0 0) 0x7f4ae8145bd0 con 0x7f4ae81457f0 2024-03-29T10:47:29.532 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.548+0000 7f4ae5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:29.533 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.548+0000 7f4ae5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:29.533 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.548+0000 7f4ae5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:29.533 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.548+0000 7f4ae5ffb640 10 monclient(hunting): my global_id is 4495 2024-03-29T10:47:29.533 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.548+0000 7f4ae5ffb640 1 -- 172.21.15.57:0/3671845110 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ae8145eb0 con 0x7f4ae81457f0 2024-03-29T10:47:29.533 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.548+0000 7f4ae5ffb640 1 -- 172.21.15.57:0/3671845110 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3812368690 0 0) 0x7f4ad0003520 con 0x7f4ae80a9d70 2024-03-29T10:47:29.533 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.548+0000 7f4ae5ffb640 1 -- 172.21.15.57:0/3671845110 >> v1:172.21.15.179:6789/0 conn(0x7f4ae81457f0 legacy=0x7f4ae813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:29.533 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.548+0000 7f4ae5ffb640 1 -- 172.21.15.57:0/3671845110 >> v1:172.21.15.57:6789/0 conn(0x7f4ae813f650 legacy=0x7f4ae80b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:29.542 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.550+0000 7f4ae5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:29.543 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.550+0000 7f4ae5ffb640 1 monclient: found mon.noname-b 2024-03-29T10:47:29.543 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.550+0000 7f4ae5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:29.543 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.550+0000 7f4ae5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:47:29.543 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.550+0000 7f4ae5ffb640 1 -- 172.21.15.57:0/3671845110 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae8146340 con 0x7f4ae80a9d70 2024-03-29T10:47:29.543 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.550+0000 7f4ae5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:29.543 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.550+0000 7f4ae5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:29.543 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.550+0000 7f4ae5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:29.543 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.550+0000 7f4aed286640 5 monclient: authenticate success, global_id 4535 2024-03-29T10:47:29.543 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.550+0000 7f4aed286640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:29.543 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.550+0000 7f4ae5ffb640 1 -- 172.21.15.57:0/3671845110 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f4ae0001a10 con 0x7f4ae80a9d70 2024-03-29T10:47:29.543 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.550+0000 7f4ae5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:29.543 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.550+0000 7f4ae5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:29.544 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.550+0000 7f4ae5ffb640 10 monclient: dump: 2024-03-29T10:47:29.544 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:29.544 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:29.544 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:29.544 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:29.544 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:29.544 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:29.544 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:29.544 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:29.544 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:29.544 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:29.544 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.550+0000 7f4aed286640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:29.544 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.550+0000 7f4ae5ffb640 1 -- 172.21.15.57:0/3671845110 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ae0001c60 con 0x7f4ae80a9d70 2024-03-29T10:47:29.544 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.550+0000 7f4ae5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:29.544 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.550+0000 7f4aed286640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:29.545 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.550+0000 7f4aed286640 4 set_mon_vals no callback set 2024-03-29T10:47:29.545 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.551+0000 7f4ae5ffb640 1 -- 172.21.15.57:0/3671845110 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f4ae0001ff0 con 0x7f4ae80a9d70 2024-03-29T10:47:29.545 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.551+0000 7f4ae5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:29.545 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.551+0000 7f4ae5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:29.545 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.551+0000 7f4ae5ffb640 10 monclient: dump: 2024-03-29T10:47:29.545 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:29.545 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:29.545 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:29.545 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:29.545 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:29.545 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:29.545 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:29.545 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:29.545 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:29.546 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:29.546 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.551+0000 7f4aed286640 10 monclient: shutdown 2024-03-29T10:47:29.546 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.551+0000 7f4aed286640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:29.546 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.551+0000 7f4aed286640 1 -- 172.21.15.57:0/3671845110 >> v1:172.21.15.57:6790/0 conn(0x7f4ae80a9d70 legacy=0x7f4ae814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:29.546 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.551+0000 7f4aed286640 1 -- 172.21.15.57:0/3671845110 shutdown_connections 2024-03-29T10:47:29.546 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.551+0000 7f4aed286640 1 -- 172.21.15.57:0/3671845110 >> 172.21.15.57:0/3671845110 conn(0x7f4ae8066a00 msgr2=0x7f4ae8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:29.546 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.551+0000 7f4aed286640 1 -- 172.21.15.57:0/3671845110 shutdown_connections 2024-03-29T10:47:29.546 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.551+0000 7f4aed286640 1 -- 172.21.15.57:0/3671845110 wait complete. 2024-03-29T10:47:29.546 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.552+0000 7f4aed286640 10 monclient: build_initial_monmap 2024-03-29T10:47:29.546 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.552+0000 7f4aed286640 10 monclient: monmap: 2024-03-29T10:47:29.546 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:47:29.546 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:29.546 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:47:29.546 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:47:29.546 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:29.547 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:29.547 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:47:29.547 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:47:29.547 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:47:29.547 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:29.547 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.553+0000 7f4aed286640 1 librados: starting msgr at 2024-03-29T10:47:29.547 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.553+0000 7f4aed286640 1 librados: starting objecter 2024-03-29T10:47:29.547 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.553+0000 7f4aed286640 1 Processor -- start 2024-03-29T10:47:29.547 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.553+0000 7f4aed286640 1 -- start start 2024-03-29T10:47:29.547 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.553+0000 7f4aed286640 1 librados: setting wanted keys 2024-03-29T10:47:29.547 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.553+0000 7f4aed286640 1 librados: calling monclient init 2024-03-29T10:47:29.547 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.553+0000 7f4aed286640 10 monclient: init 2024-03-29T10:47:29.547 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.554+0000 7f4aed286640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:29.547 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.554+0000 7f4aed286640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:29.547 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.554+0000 7f4aed286640 10 monclient(hunting): picked mon.noname-a con 0x7f4ae80a9d70 addr v1:172.21.15.57:6789/0 2024-03-29T10:47:29.548 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.554+0000 7f4aed286640 10 monclient(hunting): picked mon.noname-c con 0x7f4ae813f650 addr v1:172.21.15.57:6790/0 2024-03-29T10:47:29.548 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.554+0000 7f4aed286640 10 monclient(hunting): picked mon.noname-b con 0x7f4ae81457f0 addr v1:172.21.15.179:6789/0 2024-03-29T10:47:29.548 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.554+0000 7f4aed286640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ae80b0db0 con 0x7f4ae80a9d70 2024-03-29T10:47:29.548 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.554+0000 7f4aed286640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ae813aa40 con 0x7f4ae81457f0 2024-03-29T10:47:29.548 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.554+0000 7f4aed286640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ae813ad20 con 0x7f4ae813f650 2024-03-29T10:47:29.548 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.554+0000 7f4aed286640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:29.548 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.554+0000 7f4aed286640 10 monclient(hunting): authenticate will time out at 1401.730335s 2024-03-29T10:47:29.548 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.554+0000 7f4ae67fc640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f4ae813f650 0x7f4ae81d7670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:44326/0 (socket says 172.21.15.57:44326) 2024-03-29T10:47:29.548 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.554+0000 7f4ae67fc640 1 -- 172.21.15.57:0/2796285852 learned_addr learned my addr 172.21.15.57:0/2796285852 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:47:29.548 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.554+0000 7f4acf7fe640 10 client.?.objecter ms_handle_connect 0x7f4ae813f650 2024-03-29T10:47:29.548 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.554+0000 7f4acf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:29.548 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.554+0000 7f4acf7fe640 1 -- 172.21.15.57:0/2796285852 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178283978 0 0) 0x7f4ae813ad20 con 0x7f4ae813f650 2024-03-29T10:47:29.548 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.554+0000 7f4acf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:29.548 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.554+0000 7f4acf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:29.548 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.554+0000 7f4acf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:29.549 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.554+0000 7f4acf7fe640 10 monclient(hunting): my global_id is 4538 2024-03-29T10:47:29.549 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.554+0000 7f4acf7fe640 1 -- 172.21.15.57:0/2796285852 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ad4004eb0 con 0x7f4ae813f650 2024-03-29T10:47:29.549 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.555+0000 7f4acf7fe640 1 -- 172.21.15.57:0/2796285852 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1857613179 0 0) 0x7f4ad4004eb0 con 0x7f4ae813f650 2024-03-29T10:47:29.549 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.555+0000 7f4acf7fe640 1 -- 172.21.15.57:0/2796285852 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ae813ad20 con 0x7f4ae813f650 2024-03-29T10:47:29.549 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.555+0000 7f4acf7fe640 1 -- 172.21.15.57:0/2796285852 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f4adc001960 con 0x7f4ae813f650 2024-03-29T10:47:29.549 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.555+0000 7f4acf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:29.549 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.555+0000 7f4acf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:29.549 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.555+0000 7f4acf7fe640 10 monclient(hunting): dump: 2024-03-29T10:47:29.549 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:29.549 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:29.549 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:29.549 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:29.549 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:29.549 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:29.549 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:29.550 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:29.550 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:29.550 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:29.550 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.555+0000 7f4acf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:29.550 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.555+0000 7f4acf7fe640 1 -- 172.21.15.57:0/2796285852 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2542301645 0 0) 0x7f4ae813ad20 con 0x7f4ae813f650 2024-03-29T10:47:29.550 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.555+0000 7f4acf7fe640 1 -- 172.21.15.57:0/2796285852 >> v1:172.21.15.179:6789/0 conn(0x7f4ae81457f0 legacy=0x7f4ae81d7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:29.550 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.555+0000 7f4acf7fe640 1 -- 172.21.15.57:0/2796285852 >> v1:172.21.15.57:6789/0 conn(0x7f4ae80a9d70 legacy=0x7f4ae813b4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:47:29.550 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.557+0000 7f4acf7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:29.550 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.557+0000 7f4acf7fe640 1 monclient: found mon.c 2024-03-29T10:47:29.550 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.557+0000 7f4acf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:29.550 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.557+0000 7f4acf7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:47:29.550 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.557+0000 7f4acf7fe640 1 -- 172.21.15.57:0/2796285852 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae8142bb0 con 0x7f4ae813f650 2024-03-29T10:47:29.550 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.557+0000 7f4acf7fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:29.550 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.557+0000 7f4acf7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:29.551 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.557+0000 7f4acf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:29.551 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.557+0000 7f4acf7fe640 10 client.?.objecter ms_handle_connect 0x7f4ae81457f0 2024-03-29T10:47:29.551 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.557+0000 7f4acf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:29.551 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.557+0000 7f4aed286640 5 monclient: authenticate success, global_id 4538 2024-03-29T10:47:29.551 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.557+0000 7f4aed286640 10 monclient: _renew_subs 2024-03-29T10:47:29.551 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.557+0000 7f4aed286640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:47:29.551 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.557+0000 7f4aed286640 1 -- 172.21.15.57:0/2796285852 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ae8142e00 con 0x7f4ae813f650 2024-03-29T10:47:29.551 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.557+0000 7f4acf7fe640 1 -- 172.21.15.57:0/2796285852 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4adc001ac0 con 0x7f4ae813f650 2024-03-29T10:47:29.551 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.557+0000 7f4acf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:29.551 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.557+0000 7f4ae57fa640 4 set_mon_vals no callback set 2024-03-29T10:47:29.551 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.557+0000 7f4aed286640 10 client.4538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:29.551 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.558+0000 7f4aed286640 10 monclient: _renew_subs 2024-03-29T10:47:29.551 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.558+0000 7f4aed286640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:47:29.551 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.558+0000 7f4aed286640 1 -- 172.21.15.57:0/2796285852 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ae81e2f00 con 0x7f4ae813f650 2024-03-29T10:47:29.551 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.558+0000 7f4aed286640 1 librados: init done 2024-03-29T10:47:29.552 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.558+0000 7f4acf7fe640 1 -- 172.21.15.57:0/2796285852 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f4adc003a40 con 0x7f4ae813f650 2024-03-29T10:47:29.552 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.558+0000 7f4acf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:29.552 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.558+0000 7f4acf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:29.552 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.558+0000 7f4acf7fe640 10 monclient: dump: 2024-03-29T10:47:29.552 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:29.552 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:29.552 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:29.552 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:29.552 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:29.552 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:29.552 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:29.552 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:29.552 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:29.552 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:29.552 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.559+0000 7f4acf7fe640 1 -- 172.21.15.57:0/2796285852 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f4adc003ba0 con 0x7f4ae813f650 2024-03-29T10:47:29.553 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.560+0000 7f4aed286640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:29.553 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.560+0000 7f4aed286640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:29.553 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.560+0000 7f4aed286640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:47:29.553 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.560+0000 7f4aed286640 1 -- 172.21.15.57:0/2796285852 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ae80b0f40 con 0x7f4ae813f650 2024-03-29T10:47:29.553 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.560+0000 7f4acf7fe640 1 -- 172.21.15.57:0/2796285852 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f4adc084b40 con 0x7f4ae813f650 2024-03-29T10:47:29.553 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.560+0000 7f4acf7fe640 10 client.4538.objecter ms_dispatch 0x7f4ae8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:29.553 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.560+0000 7f4acf7fe640 3 client.4538.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:29.553 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.560+0000 7f4acf7fe640 3 client.4538.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:29.553 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.560+0000 7f4acf7fe640 20 client.4538.objecter dump_active .. 0 homeless 2024-03-29T10:47:29.553 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.560+0000 7f4ae6ffd640 10 monclient: get_auth_request con 0x7f4ad406dc20 auth_method 0 2024-03-29T10:47:29.554 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.561+0000 7f4acf7fe640 10 client.4538.objecter ms_handle_connect 0x7f4ad406dc20 2024-03-29T10:47:29.554 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.567+0000 7f4acf7fe640 1 -- 172.21.15.57:0/2796285852 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ae80b0f40 con 0x7f4ae813f650 2024-03-29T10:47:29.554 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.567+0000 7f4acf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:29.554 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.567+0000 7f4acf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:29.860 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.878+0000 7f4aed286640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:29.860 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.878+0000 7f4aed286640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:29.861 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.878+0000 7f4aed286640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:47:29.861 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.878+0000 7f4aed286640 1 -- 172.21.15.57:0/2796285852 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ae80b0f40 con 0x7f4ae813f650 2024-03-29T10:47:29.861 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.879+0000 7f4acf7fe640 1 -- 172.21.15.57:0/2796285852 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f4ae80b0f40 con 0x7f4ae813f650 2024-03-29T10:47:29.861 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.879+0000 7f4acf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:29.861 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.879+0000 7f4acf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:29.861 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:47:29.862 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:29.864 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.881+0000 7f4aed286640 10 librados: watch_flush enter 2024-03-29T10:47:29.864 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.881+0000 7f4aed286640 10 librados: watch_flush exit 2024-03-29T10:47:29.864 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.881+0000 7f4aed286640 20 client.4538.objecter shutdown clearing up homeless session... 2024-03-29T10:47:29.864 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.881+0000 7f4aed286640 10 client.4538.objecter successfully canceled tick 2024-03-29T10:47:29.864 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.881+0000 7f4aed286640 1 -- 172.21.15.57:0/2796285852 >> v1:172.21.15.179:6816/2698295936 conn(0x7f4ad406dc20 legacy=0x7f4ad40700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:29.864 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.881+0000 7f4aed286640 10 monclient: shutdown 2024-03-29T10:47:29.864 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.881+0000 7f4aed286640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:29.864 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.881+0000 7f4aed286640 1 -- 172.21.15.57:0/2796285852 >> v1:172.21.15.57:6790/0 conn(0x7f4ae813f650 legacy=0x7f4ae81d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:29.865 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.881+0000 7f4aed286640 1 -- 172.21.15.57:0/2796285852 shutdown_connections 2024-03-29T10:47:29.865 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.881+0000 7f4aed286640 1 -- 172.21.15.57:0/2796285852 >> 172.21.15.57:0/2796285852 conn(0x7f4ae8066a00 msgr2=0x7f4ae80b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:29.865 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.883+0000 7f4aed286640 1 -- 172.21.15.57:0/2796285852 shutdown_connections 2024-03-29T10:47:29.865 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.883+0000 7f4aed286640 1 -- 172.21.15.57:0/2796285852 wait complete. 2024-03-29T10:47:29.865 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:29.883+0000 7f4aed286640 1 librados: shutdown 2024-03-29T10:47:29.880 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:36.687 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:36.844 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.861+0000 7f1a1c710640 10 monclient: get_monmap_and_config 2024-03-29T10:47:36.844 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.861+0000 7f1a1c710640 10 monclient: build_initial_monmap 2024-03-29T10:47:36.844 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.861+0000 7f1a1c710640 10 monclient: monmap: 2024-03-29T10:47:36.844 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:47:36.844 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:36.844 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:47:36.861971+0000 2024-03-29T10:47:36.844 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:47:36.861971+0000 2024-03-29T10:47:36.844 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:36.844 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:36.845 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:47:36.845 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:47:36.845 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:47:36.845 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:36.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.862+0000 7f1a1c710640 1 Processor -- start 2024-03-29T10:47:36.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.862+0000 7f1a1c710640 1 -- start start 2024-03-29T10:47:36.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.862+0000 7f1a1c710640 10 monclient: init 2024-03-29T10:47:36.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.863+0000 7f1a1c710640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:36.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.863+0000 7f1a1c710640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:47:36.849 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.863+0000 7f1a1c710640 10 monclient(hunting): picked mon.noname-b con 0x7f1a140a9930 addr v1:172.21.15.57:6790/0 2024-03-29T10:47:36.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.863+0000 7f1a1c710640 10 monclient(hunting): picked mon.noname-c con 0x7f1a14149dc0 addr v1:172.21.15.179:6789/0 2024-03-29T10:47:36.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.863+0000 7f1a1c710640 10 monclient(hunting): picked mon.noname-a con 0x7f1a1414ff40 addr v1:172.21.15.57:6789/0 2024-03-29T10:47:36.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.863+0000 7f1a1c710640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a140aae60 con 0x7f1a1414ff40 2024-03-29T10:47:36.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.863+0000 7f1a1c710640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a140b4310 con 0x7f1a14149dc0 2024-03-29T10:47:36.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.863+0000 7f1a1c710640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a140afdd0 con 0x7f1a140a9930 2024-03-29T10:47:36.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.863+0000 7f1a1c710640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:36.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.863+0000 7f1a1c710640 10 monclient(hunting): authenticate will time out at 1409.039439s 2024-03-29T10:47:36.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.865+0000 7f1a1a4ac640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f1a140a9930 0x7f1a14143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:58902/0 (socket says 172.21.15.57:58902) 2024-03-29T10:47:36.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.865+0000 7f1a1a4ac640 1 -- 172.21.15.57:0/1302525103 learned_addr learned my addr 172.21.15.57:0/1302525103 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:47:36.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.865+0000 7f1a194aa640 1 -- 172.21.15.57:0/1302525103 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550100432 0 0) 0x7f1a140afdd0 con 0x7f1a140a9930 2024-03-29T10:47:36.850 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.865+0000 7f1a194aa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:36.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.865+0000 7f1a194aa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:36.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.865+0000 7f1a194aa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:36.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.865+0000 7f1a194aa640 10 monclient(hunting): my global_id is 4541 2024-03-29T10:47:36.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.866+0000 7f1a194aa640 1 -- 172.21.15.57:0/1302525103 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a00003520 con 0x7f1a140a9930 2024-03-29T10:47:36.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.866+0000 7f1a194aa640 1 -- 172.21.15.57:0/1302525103 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 610990984 0 0) 0x7f1a00003520 con 0x7f1a140a9930 2024-03-29T10:47:36.851 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.866+0000 7f1a194aa640 1 -- 172.21.15.57:0/1302525103 >> v1:172.21.15.179:6789/0 conn(0x7f1a14149dc0 legacy=0x7f1a1414a240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:36.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.867+0000 7f1a194aa640 1 -- 172.21.15.57:0/1302525103 >> v1:172.21.15.57:6789/0 conn(0x7f1a1414ff40 legacy=0x7f1a14152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:36.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.867+0000 7f1a194aa640 10 monclient: _finish_hunting 0 2024-03-29T10:47:36.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.867+0000 7f1a194aa640 1 monclient: found mon.noname-b 2024-03-29T10:47:36.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.867+0000 7f1a194aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:36.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.867+0000 7f1a194aa640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:47:36.853 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.867+0000 7f1a194aa640 1 -- 172.21.15.57:0/1302525103 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1414a950 con 0x7f1a140a9930 2024-03-29T10:47:36.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.867+0000 7f1a194aa640 10 monclient: _finish_auth 0 2024-03-29T10:47:36.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.867+0000 7f1a194aa640 10 monclient: _check_auth_tickets 2024-03-29T10:47:36.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.867+0000 7f1a194aa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:36.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.867+0000 7f1a1c710640 5 monclient: authenticate success, global_id 4541 2024-03-29T10:47:36.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.867+0000 7f1a1c710640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:36.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.867+0000 7f1a194aa640 1 -- 172.21.15.57:0/1302525103 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1a0c001a10 con 0x7f1a140a9930 2024-03-29T10:47:36.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.867+0000 7f1a194aa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:36.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.867+0000 7f1a194aa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:36.854 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.867+0000 7f1a194aa640 10 monclient: dump: 2024-03-29T10:47:36.855 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:36.855 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:36.855 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:36.855 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:36.855 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:36.855 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:36.855 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:36.855 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:36.855 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:36.855 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:36.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.867+0000 7f1a1c710640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:36.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.871+0000 7f1a194aa640 1 -- 172.21.15.57:0/1302525103 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a0c001be0 con 0x7f1a140a9930 2024-03-29T10:47:36.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.871+0000 7f1a194aa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:36.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.871+0000 7f1a1c710640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:36.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.871+0000 7f1a1c710640 4 set_mon_vals no callback set 2024-03-29T10:47:36.856 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.872+0000 7f1a194aa640 1 -- 172.21.15.57:0/1302525103 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1a0c002030 con 0x7f1a140a9930 2024-03-29T10:47:36.857 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.872+0000 7f1a194aa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:36.857 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.872+0000 7f1a194aa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:36.857 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.872+0000 7f1a194aa640 10 monclient: dump: 2024-03-29T10:47:36.857 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:36.857 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:36.857 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:36.857 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:36.857 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:36.857 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:36.857 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:36.857 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:36.857 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:36.857 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:36.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.872+0000 7f1a1c710640 10 monclient: shutdown 2024-03-29T10:47:36.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.872+0000 7f1a1c710640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:36.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.872+0000 7f1a1c710640 1 -- 172.21.15.57:0/1302525103 >> v1:172.21.15.57:6790/0 conn(0x7f1a140a9930 legacy=0x7f1a14143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:36.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.875+0000 7f1a1c710640 1 -- 172.21.15.57:0/1302525103 shutdown_connections 2024-03-29T10:47:36.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.875+0000 7f1a1c710640 1 -- 172.21.15.57:0/1302525103 >> 172.21.15.57:0/1302525103 conn(0x7f1a140b35e0 msgr2=0x7f1a140b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:36.858 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.875+0000 7f1a1c710640 1 -- 172.21.15.57:0/1302525103 shutdown_connections 2024-03-29T10:47:36.859 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.875+0000 7f1a1c710640 1 -- 172.21.15.57:0/1302525103 wait complete. 2024-03-29T10:47:36.859 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.876+0000 7f1a1c710640 10 monclient: build_initial_monmap 2024-03-29T10:47:36.859 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.876+0000 7f1a1c710640 10 monclient: monmap: 2024-03-29T10:47:36.859 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:47:36.859 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:36.859 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:47:36.859 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:47:36.860 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:36.860 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:36.860 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:47:36.860 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:47:36.860 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:47:36.860 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:36.860 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.877+0000 7f1a1c710640 1 librados: starting msgr at 2024-03-29T10:47:36.860 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.877+0000 7f1a1c710640 1 librados: starting objecter 2024-03-29T10:47:36.861 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.877+0000 7f1a1c710640 1 Processor -- start 2024-03-29T10:47:36.861 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.877+0000 7f1a1c710640 1 -- start start 2024-03-29T10:47:36.861 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.877+0000 7f1a1c710640 1 librados: setting wanted keys 2024-03-29T10:47:36.861 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.877+0000 7f1a1c710640 1 librados: calling monclient init 2024-03-29T10:47:36.861 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.877+0000 7f1a1c710640 10 monclient: init 2024-03-29T10:47:36.865 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.878+0000 7f1a1c710640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:36.866 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.878+0000 7f1a1c710640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:36.866 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.878+0000 7f1a1c710640 10 monclient(hunting): picked mon.noname-a con 0x7f1a140a9930 addr v1:172.21.15.57:6789/0 2024-03-29T10:47:36.866 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.878+0000 7f1a1c710640 10 monclient(hunting): picked mon.noname-c con 0x7f1a14149dc0 addr v1:172.21.15.57:6790/0 2024-03-29T10:47:36.866 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.878+0000 7f1a1c710640 10 monclient(hunting): picked mon.noname-b con 0x7f1a1414ff40 addr v1:172.21.15.179:6789/0 2024-03-29T10:47:36.866 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.878+0000 7f1a1c710640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a14145d80 con 0x7f1a140a9930 2024-03-29T10:47:36.866 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.878+0000 7f1a1c710640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a141e3b90 con 0x7f1a1414ff40 2024-03-29T10:47:36.866 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.878+0000 7f1a1c710640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a14144c20 con 0x7f1a14149dc0 2024-03-29T10:47:36.866 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.878+0000 7f1a1c710640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:36.866 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.878+0000 7f1a1c710640 10 monclient(hunting): authenticate will time out at 1409.054573s 2024-03-29T10:47:36.873 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.884+0000 7f1a1a4ac640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1a140a9930 0x7f1a14145670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:58397/0 (socket says 172.21.15.57:58397) 2024-03-29T10:47:36.874 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.884+0000 7f1a1a4ac640 1 -- 172.21.15.57:0/3852161775 learned_addr learned my addr 172.21.15.57:0/3852161775 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:47:36.874 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.885+0000 7f1a0affd640 10 client.?.objecter ms_handle_connect 0x7f1a140a9930 2024-03-29T10:47:36.874 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.885+0000 7f1a0affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:36.874 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.885+0000 7f1a0affd640 10 client.?.objecter ms_handle_connect 0x7f1a14149dc0 2024-03-29T10:47:36.874 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.885+0000 7f1a0affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:36.874 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.885+0000 7f1a0affd640 10 client.?.objecter ms_handle_connect 0x7f1a1414ff40 2024-03-29T10:47:36.874 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.885+0000 7f1a0affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:36.874 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.885+0000 7f1a0affd640 1 -- 172.21.15.57:0/3852161775 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738818146 0 0) 0x7f1a14145d80 con 0x7f1a140a9930 2024-03-29T10:47:36.874 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.885+0000 7f1a0affd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:36.874 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.885+0000 7f1a0affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:36.874 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.885+0000 7f1a0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:36.874 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.885+0000 7f1a0affd640 10 monclient(hunting): my global_id is 4494 2024-03-29T10:47:36.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.885+0000 7f1a0affd640 1 -- 172.21.15.57:0/3852161775 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19f8004eb0 con 0x7f1a140a9930 2024-03-29T10:47:36.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.885+0000 7f1a0affd640 1 -- 172.21.15.57:0/3852161775 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981696799 0 0) 0x7f1a141e3b90 con 0x7f1a1414ff40 2024-03-29T10:47:36.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.885+0000 7f1a0affd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:36.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.885+0000 7f1a0affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:36.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.885+0000 7f1a0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:36.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.885+0000 7f1a0affd640 10 monclient(hunting): my global_id is 4498 2024-03-29T10:47:36.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.885+0000 7f1a0affd640 1 -- 172.21.15.57:0/3852161775 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a14145d80 con 0x7f1a1414ff40 2024-03-29T10:47:36.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a0affd640 1 -- 172.21.15.57:0/3852161775 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977479439 0 0) 0x7f1a14144c20 con 0x7f1a14149dc0 2024-03-29T10:47:36.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a0affd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:36.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a0affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:36.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:36.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a0affd640 10 monclient(hunting): my global_id is 4544 2024-03-29T10:47:36.875 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a0affd640 1 -- 172.21.15.57:0/3852161775 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a141e3b90 con 0x7f1a14149dc0 2024-03-29T10:47:36.876 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a0affd640 1 -- 172.21.15.57:0/3852161775 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3667113785 0 0) 0x7f19f8004eb0 con 0x7f1a140a9930 2024-03-29T10:47:36.876 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a0affd640 1 -- 172.21.15.57:0/3852161775 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19f8005620 con 0x7f1a140a9930 2024-03-29T10:47:36.876 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a0affd640 1 -- 172.21.15.57:0/3852161775 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1a0c001ef0 con 0x7f1a140a9930 2024-03-29T10:47:36.876 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a0affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:36.876 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a0affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:36.876 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a0affd640 10 monclient(hunting): dump: 2024-03-29T10:47:36.876 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:36.876 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:36.876 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:36.876 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:36.876 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:36.876 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:36.877 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:36.877 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:36.877 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:36.877 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:36.877 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a0affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:36.877 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a0affd640 1 -- 172.21.15.57:0/3852161775 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 919064897 0 0) 0x7f19f8005620 con 0x7f1a140a9930 2024-03-29T10:47:36.877 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a0affd640 1 -- 172.21.15.57:0/3852161775 >> v1:172.21.15.57:6790/0 conn(0x7f1a14149dc0 legacy=0x7f1a141e0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:36.877 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a0affd640 1 -- 172.21.15.57:0/3852161775 >> v1:172.21.15.179:6789/0 conn(0x7f1a1414ff40 legacy=0x7f1a141e4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:36.877 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a0affd640 10 monclient: _finish_hunting 0 2024-03-29T10:47:36.877 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a0affd640 1 monclient: found mon.a 2024-03-29T10:47:36.877 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:36.877 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a0affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:47:36.877 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a0affd640 1 -- 172.21.15.57:0/3852161775 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a14144510 con 0x7f1a140a9930 2024-03-29T10:47:36.877 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a0affd640 10 monclient: _finish_auth 0 2024-03-29T10:47:36.878 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a0affd640 10 monclient: _check_auth_tickets 2024-03-29T10:47:36.878 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:36.878 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a1c710640 5 monclient: authenticate success, global_id 4494 2024-03-29T10:47:36.878 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a1c710640 10 monclient: _renew_subs 2024-03-29T10:47:36.878 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a1c710640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:47:36.878 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a1c710640 1 -- 172.21.15.57:0/3852161775 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a1413f090 con 0x7f1a140a9930 2024-03-29T10:47:36.878 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a1c710640 10 client.4494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:36.878 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a1c710640 10 monclient: _renew_subs 2024-03-29T10:47:36.878 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a1c710640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:47:36.878 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a1c710640 1 -- 172.21.15.57:0/3852161775 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a141e14c0 con 0x7f1a140a9930 2024-03-29T10:47:36.878 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.886+0000 7f1a1c710640 1 librados: init done 2024-03-29T10:47:36.878 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.889+0000 7f1a1c710640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:36.879 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.889+0000 7f1a1c710640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:36.879 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.889+0000 7f1a1c710640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:47:36.879 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.890+0000 7f1a1c710640 1 -- 172.21.15.57:0/3852161775 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a141e17c0 con 0x7f1a140a9930 2024-03-29T10:47:36.879 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.890+0000 7f1a0affd640 1 -- 172.21.15.57:0/3852161775 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a0c004f70 con 0x7f1a140a9930 2024-03-29T10:47:36.879 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.890+0000 7f1a0affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:36.879 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.890+0000 7f1a0affd640 1 -- 172.21.15.57:0/3852161775 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1a0c005280 con 0x7f1a140a9930 2024-03-29T10:47:36.879 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.890+0000 7f1a0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:36.879 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.890+0000 7f1a0affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:36.879 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.890+0000 7f1a0affd640 10 monclient: dump: 2024-03-29T10:47:36.879 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:36.879 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:36.879 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:36.880 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:36.880 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:36.880 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:36.880 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:36.880 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:36.880 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:36.880 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:36.880 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.890+0000 7f1a0affd640 1 -- 172.21.15.57:0/3852161775 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f1a0c01b250 con 0x7f1a140a9930 2024-03-29T10:47:36.880 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.891+0000 7f1a0affd640 1 -- 172.21.15.57:0/3852161775 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f1a0c085000 con 0x7f1a140a9930 2024-03-29T10:47:36.880 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.891+0000 7f1a0affd640 10 client.4494.objecter ms_dispatch 0x7f1a14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:36.880 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.891+0000 7f1a0affd640 3 client.4494.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:36.881 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.891+0000 7f1a0affd640 3 client.4494.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:36.881 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.891+0000 7f1a0affd640 20 client.4494.objecter dump_active .. 0 homeless 2024-03-29T10:47:36.881 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.890+0000 7f1a18ca9640 4 set_mon_vals no callback set 2024-03-29T10:47:36.881 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.892+0000 7f1a19cab640 10 monclient: get_auth_request con 0x7f19f806e080 auth_method 0 2024-03-29T10:47:36.881 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.892+0000 7f1a0affd640 10 client.4494.objecter ms_handle_connect 0x7f19f806e080 2024-03-29T10:47:36.882 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.900+0000 7f1a0affd640 1 -- 172.21.15.57:0/3852161775 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1a0c050960 con 0x7f1a140a9930 2024-03-29T10:47:36.882 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.900+0000 7f1a0affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:36.883 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:36.900+0000 7f1a0affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:37.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:37.243+0000 7f1a1c710640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:37.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:37.243+0000 7f1a1c710640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:37.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:37.243+0000 7f1a1c710640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:47:37.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:37.243+0000 7f1a1c710640 1 -- 172.21.15.57:0/3852161775 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a14075f10 con 0x7f1a140a9930 2024-03-29T10:47:37.229 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:37.246+0000 7f1a0affd640 1 -- 172.21.15.57:0/3852161775 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f1a0c052890 con 0x7f1a140a9930 2024-03-29T10:47:37.229 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:37.247+0000 7f1a0affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:37.229 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:37.247+0000 7f1a0affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:37.232 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:47:37.232 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:37.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:37.250+0000 7f1a1c710640 10 librados: watch_flush enter 2024-03-29T10:47:37.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:37.250+0000 7f1a1c710640 10 librados: watch_flush exit 2024-03-29T10:47:37.233 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:37.250+0000 7f1a1c710640 20 client.4494.objecter shutdown clearing up homeless session... 2024-03-29T10:47:37.233 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:37.250+0000 7f1a1c710640 10 client.4494.objecter successfully canceled tick 2024-03-29T10:47:37.234 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:37.251+0000 7f1a1c710640 1 -- 172.21.15.57:0/3852161775 >> v1:172.21.15.179:6816/2698295936 conn(0x7f19f806e080 legacy=0x7f19f8070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:37.234 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:37.251+0000 7f1a1c710640 10 monclient: shutdown 2024-03-29T10:47:37.234 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:37.251+0000 7f1a1c710640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:37.234 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:37.251+0000 7f1a1c710640 1 -- 172.21.15.57:0/3852161775 >> v1:172.21.15.57:6789/0 conn(0x7f1a140a9930 legacy=0x7f1a14145670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:37.234 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:37.251+0000 7f1a1c710640 1 -- 172.21.15.57:0/3852161775 shutdown_connections 2024-03-29T10:47:37.234 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:37.251+0000 7f1a1c710640 1 -- 172.21.15.57:0/3852161775 >> 172.21.15.57:0/3852161775 conn(0x7f1a140b35e0 msgr2=0x7f1a140b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:37.234 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:37.252+0000 7f1a1c710640 1 -- 172.21.15.57:0/3852161775 shutdown_connections 2024-03-29T10:47:37.234 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:37.252+0000 7f1a1c710640 1 -- 172.21.15.57:0/3852161775 wait complete. 2024-03-29T10:47:37.234 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:37.252+0000 7f1a1c710640 1 librados: shutdown 2024-03-29T10:47:37.251 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:44.059 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:44.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.230+0000 7f4c1f2b3640 10 monclient: get_monmap_and_config 2024-03-29T10:47:44.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.230+0000 7f4c1f2b3640 10 monclient: build_initial_monmap 2024-03-29T10:47:44.213 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.231+0000 7f4c1f2b3640 10 monclient: monmap: 2024-03-29T10:47:44.213 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:47:44.213 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:44.213 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:47:44.231705+0000 2024-03-29T10:47:44.214 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:47:44.231705+0000 2024-03-29T10:47:44.214 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:44.214 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:44.214 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:47:44.214 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:47:44.214 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:47:44.214 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:44.217 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.233+0000 7f4c1f2b3640 1 Processor -- start 2024-03-29T10:47:44.217 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.235+0000 7f4c1f2b3640 1 -- start start 2024-03-29T10:47:44.217 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.235+0000 7f4c1f2b3640 10 monclient: init 2024-03-29T10:47:44.218 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.236+0000 7f4c1f2b3640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:44.218 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.236+0000 7f4c1f2b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:44.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.239+0000 7f4c1f2b3640 10 monclient(hunting): picked mon.noname-a con 0x7f4c10006a70 addr v1:172.21.15.57:6789/0 2024-03-29T10:47:44.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.239+0000 7f4c1f2b3640 10 monclient(hunting): picked mon.noname-c con 0x7f4c1002b100 addr v1:172.21.15.179:6789/0 2024-03-29T10:47:44.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.239+0000 7f4c1f2b3640 10 monclient(hunting): picked mon.noname-b con 0x7f4c10029c30 addr v1:172.21.15.57:6790/0 2024-03-29T10:47:44.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.239+0000 7f4c1f2b3640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c1002a270 con 0x7f4c10006a70 2024-03-29T10:47:44.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.239+0000 7f4c1f2b3640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c1002c440 con 0x7f4c1002b100 2024-03-29T10:47:44.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.239+0000 7f4c1f2b3640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c100c9440 con 0x7f4c10029c30 2024-03-29T10:47:44.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.239+0000 7f4c1f2b3640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:44.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.239+0000 7f4c1f2b3640 10 monclient(hunting): authenticate will time out at 1416.415802s 2024-03-29T10:47:44.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.239+0000 7f4c1e2b1640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f4c10006a70 0x7f4c100c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:55085/0 (socket says 172.21.15.57:55085) 2024-03-29T10:47:44.222 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.240+0000 7f4c1e2b1640 1 -- 172.21.15.57:0/569106094 learned_addr learned my addr 172.21.15.57:0/569106094 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:47:44.223 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.241+0000 7f4c1d2af640 1 -- 172.21.15.57:0/569106094 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1457895725 0 0) 0x7f4c1002a270 con 0x7f4c10006a70 2024-03-29T10:47:44.223 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.241+0000 7f4c1d2af640 10 monclient(hunting): _negotiate 2024-03-29T10:47:44.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.241+0000 7f4c1d2af640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:44.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.241+0000 7f4c1d2af640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:44.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.241+0000 7f4c1d2af640 10 monclient(hunting): my global_id is 4497 2024-03-29T10:47:44.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.241+0000 7f4c1d2af640 1 -- 172.21.15.57:0/569106094 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c04003520 con 0x7f4c10006a70 2024-03-29T10:47:44.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.241+0000 7f4c1d2af640 1 -- 172.21.15.57:0/569106094 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1558160500 0 0) 0x7f4c100c9440 con 0x7f4c10029c30 2024-03-29T10:47:44.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.241+0000 7f4c1d2af640 10 monclient(hunting): _negotiate 2024-03-29T10:47:44.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.241+0000 7f4c1d2af640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:44.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.241+0000 7f4c1d2af640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:44.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.241+0000 7f4c1d2af640 10 monclient(hunting): my global_id is 4547 2024-03-29T10:47:44.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.241+0000 7f4c1d2af640 1 -- 172.21.15.57:0/569106094 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c1002a270 con 0x7f4c10029c30 2024-03-29T10:47:44.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.241+0000 7f4c1d2af640 1 -- 172.21.15.57:0/569106094 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3971084587 0 0) 0x7f4c1002a270 con 0x7f4c10029c30 2024-03-29T10:47:44.225 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.241+0000 7f4c1d2af640 1 -- 172.21.15.57:0/569106094 >> v1:172.21.15.179:6789/0 conn(0x7f4c1002b100 legacy=0x7f4c100cf100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:47:44.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.243+0000 7f4c1d2af640 1 -- 172.21.15.57:0/569106094 >> v1:172.21.15.57:6789/0 conn(0x7f4c10006a70 legacy=0x7f4c100c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:44.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.243+0000 7f4c1d2af640 10 monclient: _finish_hunting 0 2024-03-29T10:47:44.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.243+0000 7f4c1d2af640 1 monclient: found mon.noname-b 2024-03-29T10:47:44.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.243+0000 7f4c1d2af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:44.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.243+0000 7f4c1d2af640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:47:44.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.243+0000 7f4c1d2af640 1 -- 172.21.15.57:0/569106094 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c100c95d0 con 0x7f4c10029c30 2024-03-29T10:47:44.226 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.243+0000 7f4c1d2af640 10 monclient: _finish_auth 0 2024-03-29T10:47:44.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.243+0000 7f4c1d2af640 10 monclient: _check_auth_tickets 2024-03-29T10:47:44.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.243+0000 7f4c1d2af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:44.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.243+0000 7f4c1d2af640 1 -- 172.21.15.57:0/569106094 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f4c1804f730 con 0x7f4c10029c30 2024-03-29T10:47:44.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.243+0000 7f4c1d2af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:44.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.243+0000 7f4c1d2af640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:44.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.243+0000 7f4c1d2af640 10 monclient: dump: 2024-03-29T10:47:44.227 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:44.227 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:44.227 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:44.227 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:44.227 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:44.227 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:44.227 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:44.227 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:44.228 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:44.228 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:44.228 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.244+0000 7f4c1f2b3640 5 monclient: authenticate success, global_id 4547 2024-03-29T10:47:44.228 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.244+0000 7f4c1f2b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:47:44.228 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.244+0000 7f4c1d2af640 1 -- 172.21.15.57:0/569106094 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c180008d0 con 0x7f4c10029c30 2024-03-29T10:47:44.228 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.244+0000 7f4c1d2af640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:44.228 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.244+0000 7f4c1d2af640 1 -- 172.21.15.57:0/569106094 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f4c180758b0 con 0x7f4c10029c30 2024-03-29T10:47:44.228 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.244+0000 7f4c1d2af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:44.228 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.244+0000 7f4c1d2af640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:44.228 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.244+0000 7f4c1d2af640 10 monclient: dump: 2024-03-29T10:47:44.228 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:44.228 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:44.228 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:44.228 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:44.228 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:44.229 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:44.229 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:44.229 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:44.229 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:44.229 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:44.229 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.244+0000 7f4c1f2b3640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:44.229 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.244+0000 7f4c1f2b3640 4 set_mon_vals no callback set 2024-03-29T10:47:44.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.245+0000 7f4c1f2b3640 10 monclient: shutdown 2024-03-29T10:47:44.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.245+0000 7f4c1f2b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:44.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.245+0000 7f4c1f2b3640 1 -- 172.21.15.57:0/569106094 >> v1:172.21.15.57:6790/0 conn(0x7f4c10029c30 legacy=0x7f4c100d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:44.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.245+0000 7f4c1f2b3640 1 -- 172.21.15.57:0/569106094 shutdown_connections 2024-03-29T10:47:44.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.245+0000 7f4c1f2b3640 1 -- 172.21.15.57:0/569106094 >> 172.21.15.57:0/569106094 conn(0x7f4c100058f0 msgr2=0x7f4c10005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:44.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.247+0000 7f4c1f2b3640 1 -- 172.21.15.57:0/569106094 shutdown_connections 2024-03-29T10:47:44.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.247+0000 7f4c1f2b3640 1 -- 172.21.15.57:0/569106094 wait complete. 2024-03-29T10:47:44.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.248+0000 7f4c1f2b3640 10 monclient: build_initial_monmap 2024-03-29T10:47:44.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.248+0000 7f4c1f2b3640 10 monclient: monmap: 2024-03-29T10:47:44.231 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:47:44.231 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:44.232 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:47:44.232 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:47:44.232 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:44.232 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:44.232 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:47:44.232 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:47:44.232 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:47:44.232 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:44.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.248+0000 7f4c1f2b3640 1 librados: starting msgr at 2024-03-29T10:47:44.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.248+0000 7f4c1f2b3640 1 librados: starting objecter 2024-03-29T10:47:44.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.248+0000 7f4c1f2b3640 1 Processor -- start 2024-03-29T10:47:44.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.248+0000 7f4c1f2b3640 1 -- start start 2024-03-29T10:47:44.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.248+0000 7f4c1f2b3640 1 librados: setting wanted keys 2024-03-29T10:47:44.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.248+0000 7f4c1f2b3640 1 librados: calling monclient init 2024-03-29T10:47:44.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.248+0000 7f4c1f2b3640 10 monclient: init 2024-03-29T10:47:44.234 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.249+0000 7f4c1f2b3640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:44.234 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.249+0000 7f4c1f2b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:44.234 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.250+0000 7f4c1f2b3640 10 monclient(hunting): picked mon.noname-a con 0x7f4c10006a70 addr v1:172.21.15.57:6789/0 2024-03-29T10:47:44.234 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.250+0000 7f4c1f2b3640 10 monclient(hunting): picked mon.noname-b con 0x7f4c10029c30 addr v1:172.21.15.179:6789/0 2024-03-29T10:47:44.234 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.250+0000 7f4c1f2b3640 10 monclient(hunting): picked mon.noname-c con 0x7f4c100cd030 addr v1:172.21.15.57:6790/0 2024-03-29T10:47:44.234 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.250+0000 7f4c1f2b3640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c1002f340 con 0x7f4c10006a70 2024-03-29T10:47:44.234 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.250+0000 7f4c1f2b3640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c1002df20 con 0x7f4c10029c30 2024-03-29T10:47:44.235 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.250+0000 7f4c1f2b3640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c101618b0 con 0x7f4c100cd030 2024-03-29T10:47:44.235 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.250+0000 7f4c1f2b3640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:44.235 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.250+0000 7f4c1f2b3640 10 monclient(hunting): authenticate will time out at 1416.426117s 2024-03-29T10:47:44.235 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.251+0000 7f4c1eab2640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f4c100cd030 0x7f4c100c2780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:59048/0 (socket says 172.21.15.57:59048) 2024-03-29T10:47:44.235 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.251+0000 7f4c1eab2640 1 -- 172.21.15.57:0/4169451936 learned_addr learned my addr 172.21.15.57:0/4169451936 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:47:44.237 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.252+0000 7f4c0effd640 10 client.?.objecter ms_handle_connect 0x7f4c100cd030 2024-03-29T10:47:44.238 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.253+0000 7f4c0effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:44.238 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.253+0000 7f4c0effd640 10 client.?.objecter ms_handle_connect 0x7f4c10006a70 2024-03-29T10:47:44.238 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.253+0000 7f4c0effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:44.238 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.253+0000 7f4c0effd640 1 -- 172.21.15.57:0/4169451936 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721758715 0 0) 0x7f4c101618b0 con 0x7f4c100cd030 2024-03-29T10:47:44.238 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.253+0000 7f4c0effd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:44.238 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.253+0000 7f4c0effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:44.238 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.253+0000 7f4c0effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:44.238 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.253+0000 7f4c0effd640 10 monclient(hunting): my global_id is 4550 2024-03-29T10:47:44.238 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.253+0000 7f4c0effd640 1 -- 172.21.15.57:0/4169451936 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bfc004eb0 con 0x7f4c100cd030 2024-03-29T10:47:44.238 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.253+0000 7f4c0effd640 1 -- 172.21.15.57:0/4169451936 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131845606 0 0) 0x7f4c1002f340 con 0x7f4c10006a70 2024-03-29T10:47:44.238 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.253+0000 7f4c0effd640 10 monclient(hunting): _negotiate 2024-03-29T10:47:44.238 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.253+0000 7f4c0effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:44.239 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.253+0000 7f4c0effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:44.239 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.253+0000 7f4c0effd640 10 monclient(hunting): my global_id is 4500 2024-03-29T10:47:44.239 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.253+0000 7f4c0effd640 1 -- 172.21.15.57:0/4169451936 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c101618b0 con 0x7f4c10006a70 2024-03-29T10:47:44.239 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.253+0000 7f4c0effd640 1 -- 172.21.15.57:0/4169451936 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2987660642 0 0) 0x7f4bfc004eb0 con 0x7f4c100cd030 2024-03-29T10:47:44.239 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.253+0000 7f4c0effd640 1 -- 172.21.15.57:0/4169451936 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c1002f340 con 0x7f4c100cd030 2024-03-29T10:47:44.239 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.253+0000 7f4c0effd640 1 -- 172.21.15.57:0/4169451936 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f4c1804f400 con 0x7f4c100cd030 2024-03-29T10:47:44.239 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.253+0000 7f4c0effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:44.239 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.253+0000 7f4c0effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:44.239 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.253+0000 7f4c0effd640 10 monclient(hunting): dump: 2024-03-29T10:47:44.239 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:44.239 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:44.239 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:44.239 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:44.239 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:44.239 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:44.240 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:44.240 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:44.240 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:44.240 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:44.240 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.253+0000 7f4c0effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:44.240 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.254+0000 7f4c0effd640 1 -- 172.21.15.57:0/4169451936 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2061376675 0 0) 0x7f4c101618b0 con 0x7f4c10006a70 2024-03-29T10:47:44.240 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.254+0000 7f4c0effd640 1 -- 172.21.15.57:0/4169451936 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bfc004eb0 con 0x7f4c10006a70 2024-03-29T10:47:44.240 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.254+0000 7f4c0effd640 1 -- 172.21.15.57:0/4169451936 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f4c14001890 con 0x7f4c10006a70 2024-03-29T10:47:44.240 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.254+0000 7f4c0effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:44.240 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.254+0000 7f4c0effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:44.240 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.254+0000 7f4c0effd640 10 monclient(hunting): dump: 2024-03-29T10:47:44.240 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:44.240 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:44.240 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:44.240 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:44.241 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:44.241 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:44.241 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:44.241 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:44.241 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:44.241 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:44.241 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.254+0000 7f4c0effd640 1 -- 172.21.15.57:0/4169451936 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3617433322 0 0) 0x7f4bfc004eb0 con 0x7f4c10006a70 2024-03-29T10:47:44.241 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.254+0000 7f4c0effd640 1 -- 172.21.15.57:0/4169451936 >> v1:172.21.15.57:6790/0 conn(0x7f4c100cd030 legacy=0x7f4c100c2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:44.241 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.254+0000 7f4c0effd640 1 -- 172.21.15.57:0/4169451936 >> v1:172.21.15.179:6789/0 conn(0x7f4c10029c30 legacy=0x7f4c101611a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:44.241 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.254+0000 7f4c0effd640 10 monclient: _finish_hunting 0 2024-03-29T10:47:44.241 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.254+0000 7f4c0effd640 1 monclient: found mon.a 2024-03-29T10:47:44.241 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.254+0000 7f4c0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:44.241 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.254+0000 7f4c0effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:47:44.241 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.254+0000 7f4c0effd640 1 -- 172.21.15.57:0/4169451936 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c10161a40 con 0x7f4c10006a70 2024-03-29T10:47:44.241 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.255+0000 7f4c0effd640 10 monclient: _finish_auth 0 2024-03-29T10:47:44.242 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.255+0000 7f4c0effd640 10 monclient: _check_auth_tickets 2024-03-29T10:47:44.242 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.255+0000 7f4c0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:44.242 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.255+0000 7f4c1f2b3640 5 monclient: authenticate success, global_id 4500 2024-03-29T10:47:44.242 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.255+0000 7f4c1f2b3640 10 monclient: _renew_subs 2024-03-29T10:47:44.242 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.255+0000 7f4c1f2b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:47:44.242 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.255+0000 7f4c1f2b3640 1 -- 172.21.15.57:0/4169451936 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c100c2e90 con 0x7f4c10006a70 2024-03-29T10:47:44.242 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.255+0000 7f4c1f2b3640 10 client.4500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:44.242 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.255+0000 7f4c1f2b3640 10 monclient: _renew_subs 2024-03-29T10:47:44.242 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.255+0000 7f4c1f2b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:47:44.242 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.255+0000 7f4c1f2b3640 1 -- 172.21.15.57:0/4169451936 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c1002d230 con 0x7f4c10006a70 2024-03-29T10:47:44.242 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.255+0000 7f4c1f2b3640 1 librados: init done 2024-03-29T10:47:44.244 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.256+0000 7f4c0cff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:44.244 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.256+0000 7f4c0cff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:44.244 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.256+0000 7f4c0cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:47:44.245 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.256+0000 7f4c0cff9640 1 -- 172.21.15.57:0/4169451936 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c10160ae0 con 0x7f4c10006a70 2024-03-29T10:47:44.245 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.258+0000 7f4c0effd640 1 -- 172.21.15.57:0/4169451936 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c140047e0 con 0x7f4c10006a70 2024-03-29T10:47:44.245 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.258+0000 7f4c0effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:44.245 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.259+0000 7f4c1caae640 4 set_mon_vals no callback set 2024-03-29T10:47:44.245 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.259+0000 7f4c0effd640 1 -- 172.21.15.57:0/4169451936 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f4c14004af0 con 0x7f4c10006a70 2024-03-29T10:47:44.245 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.259+0000 7f4c0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:44.245 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.259+0000 7f4c0effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:44.245 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.259+0000 7f4c0effd640 10 monclient: dump: 2024-03-29T10:47:44.245 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:44.245 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:44.245 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:44.245 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:44.245 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:44.245 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:44.245 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:44.246 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:44.246 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:44.246 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:44.246 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.259+0000 7f4c0effd640 1 -- 172.21.15.57:0/4169451936 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f4c1401aac0 con 0x7f4c10006a70 2024-03-29T10:47:44.246 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.260+0000 7f4c0effd640 1 -- 172.21.15.57:0/4169451936 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f4c140849f0 con 0x7f4c10006a70 2024-03-29T10:47:44.246 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.260+0000 7f4c0effd640 10 client.4500.objecter ms_dispatch 0x7f4c18000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:44.246 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.260+0000 7f4c0effd640 3 client.4500.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:44.246 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.260+0000 7f4c0effd640 3 client.4500.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:44.246 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.260+0000 7f4c0effd640 20 client.4500.objecter dump_active .. 0 homeless 2024-03-29T10:47:44.246 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.262+0000 7f4c1dab0640 10 monclient: get_auth_request con 0x7f4bfc06dd50 auth_method 0 2024-03-29T10:47:44.246 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.263+0000 7f4c0effd640 10 client.4500.objecter ms_handle_connect 0x7f4bfc06dd50 2024-03-29T10:47:44.248 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.266+0000 7f4c0effd640 1 -- 172.21.15.57:0/4169451936 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c14050350 con 0x7f4c10006a70 2024-03-29T10:47:44.248 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.266+0000 7f4c0effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:44.248 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.266+0000 7f4c0effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:44.579 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.597+0000 7f4c0cff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:44.579 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.597+0000 7f4c0cff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:44.579 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.597+0000 7f4c0cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:47:44.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.597+0000 7f4c0cff9640 1 -- 172.21.15.57:0/4169451936 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c10004cd0 con 0x7f4c10006a70 2024-03-29T10:47:44.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.598+0000 7f4c0effd640 1 -- 172.21.15.57:0/4169451936 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f4c14052280 con 0x7f4c10006a70 2024-03-29T10:47:44.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.598+0000 7f4c0effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:44.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.598+0000 7f4c0effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:44.581 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:47:44.581 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:44.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.601+0000 7f4c0cff9640 10 librados: watch_flush enter 2024-03-29T10:47:44.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.601+0000 7f4c0cff9640 10 librados: watch_flush exit 2024-03-29T10:47:44.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.601+0000 7f4c0cff9640 20 client.4500.objecter shutdown clearing up homeless session... 2024-03-29T10:47:44.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.601+0000 7f4c0cff9640 10 client.4500.objecter successfully canceled tick 2024-03-29T10:47:44.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.602+0000 7f4c0cff9640 1 -- 172.21.15.57:0/4169451936 >> v1:172.21.15.179:6816/2698295936 conn(0x7f4bfc06dd50 legacy=0x7f4bfc070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:44.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.602+0000 7f4c0cff9640 10 monclient: shutdown 2024-03-29T10:47:44.584 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.602+0000 7f4c0cff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:44.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.602+0000 7f4c0cff9640 1 -- 172.21.15.57:0/4169451936 >> v1:172.21.15.57:6789/0 conn(0x7f4c10006a70 legacy=0x7f4c1002d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:44.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.602+0000 7f4c0cff9640 1 -- 172.21.15.57:0/4169451936 shutdown_connections 2024-03-29T10:47:44.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.602+0000 7f4c0cff9640 1 -- 172.21.15.57:0/4169451936 >> 172.21.15.57:0/4169451936 conn(0x7f4c100058f0 msgr2=0x7f4c10005cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:44.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.603+0000 7f4c0cff9640 1 -- 172.21.15.57:0/4169451936 shutdown_connections 2024-03-29T10:47:44.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.603+0000 7f4c0cff9640 1 -- 172.21.15.57:0/4169451936 wait complete. 2024-03-29T10:47:44.585 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:44.603+0000 7f4c0cff9640 1 librados: shutdown 2024-03-29T10:47:44.603 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:51.410 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:51.561 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.577+0000 7f5af2df4640 10 monclient: get_monmap_and_config 2024-03-29T10:47:51.561 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.577+0000 7f5af2df4640 10 monclient: build_initial_monmap 2024-03-29T10:47:51.561 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.577+0000 7f5af2df4640 10 monclient: monmap: 2024-03-29T10:47:51.561 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:47:51.561 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:51.561 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:47:51.578882+0000 2024-03-29T10:47:51.561 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:47:51.578882+0000 2024-03-29T10:47:51.561 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:51.561 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:51.561 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:47:51.561 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:47:51.561 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:47:51.562 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:51.562 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.579+0000 7f5af2df4640 1 Processor -- start 2024-03-29T10:47:51.562 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.580+0000 7f5af2df4640 1 -- start start 2024-03-29T10:47:51.562 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.580+0000 7f5af2df4640 10 monclient: init 2024-03-29T10:47:51.565 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.580+0000 7f5af2df4640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:51.565 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.580+0000 7f5af2df4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:47:51.566 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.580+0000 7f5af2df4640 10 monclient(hunting): picked mon.noname-b con 0x7f5aec0b4100 addr v1:172.21.15.57:6790/0 2024-03-29T10:47:51.566 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.580+0000 7f5af2df4640 10 monclient(hunting): picked mon.noname-c con 0x7f5aec13ee90 addr v1:172.21.15.179:6789/0 2024-03-29T10:47:51.566 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.580+0000 7f5af2df4640 10 monclient(hunting): picked mon.noname-a con 0x7f5aec0af900 addr v1:172.21.15.57:6789/0 2024-03-29T10:47:51.566 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.580+0000 7f5af2df4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec0afd10 con 0x7f5aec0af900 2024-03-29T10:47:51.566 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.580+0000 7f5af2df4640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec0af6f0 con 0x7f5aec13ee90 2024-03-29T10:47:51.566 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.580+0000 7f5af2df4640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec0aaf60 con 0x7f5aec0b4100 2024-03-29T10:47:51.566 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.580+0000 7f5af2df4640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:51.566 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.580+0000 7f5af2df4640 10 monclient(hunting): authenticate will time out at 1423.757981s 2024-03-29T10:47:51.567 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.583+0000 7f5af1391640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f5aec0af900 0x7f5aec13b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:48771/0 (socket says 172.21.15.57:48771) 2024-03-29T10:47:51.567 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.583+0000 7f5af1391640 1 -- 172.21.15.57:0/4151541429 learned_addr learned my addr 172.21.15.57:0/4151541429 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:47:51.571 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.585+0000 7f5aeb7fe640 1 -- 172.21.15.57:0/4151541429 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4064638229 0 0) 0x7f5aec0afd10 con 0x7f5aec0af900 2024-03-29T10:47:51.571 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5aeb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:51.571 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5aeb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:51.571 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5aeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:51.571 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5aeb7fe640 10 monclient(hunting): my global_id is 4503 2024-03-29T10:47:51.571 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5aeb7fe640 1 -- 172.21.15.57:0/4151541429 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5adc003520 con 0x7f5aec0af900 2024-03-29T10:47:51.571 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5aeb7fe640 1 -- 172.21.15.57:0/4151541429 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2786654691 0 0) 0x7f5adc003520 con 0x7f5aec0af900 2024-03-29T10:47:51.571 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5aeb7fe640 1 -- 172.21.15.57:0/4151541429 >> v1:172.21.15.57:6790/0 conn(0x7f5aec0b4100 legacy=0x7f5aec0a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.571 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5aeb7fe640 1 -- 172.21.15.57:0/4151541429 >> v1:172.21.15.179:6789/0 conn(0x7f5aec13ee90 legacy=0x7f5aec0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:47:51.571 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5aeb7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:51.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5aeb7fe640 1 monclient: found mon.noname-a 2024-03-29T10:47:51.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5aeb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:51.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5aeb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:47:51.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5aeb7fe640 1 -- 172.21.15.57:0/4151541429 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec13e970 con 0x7f5aec0af900 2024-03-29T10:47:51.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5aeb7fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:51.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5aeb7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:51.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5aeb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:51.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5aeb7fe640 1 -- 172.21.15.57:0/4151541429 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f5ae4001690 con 0x7f5aec0af900 2024-03-29T10:47:51.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5aeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5aeb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:47:51.572 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5aeb7fe640 10 monclient: dump: 2024-03-29T10:47:51.573 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:51.573 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:51.573 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:51.573 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:51.573 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.573 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:51.573 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:51.573 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:51.573 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:51.573 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:51.574 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5aeb7fe640 1 -- 172.21.15.57:0/4151541429 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ae4001920 con 0x7f5aec0af900 2024-03-29T10:47:51.574 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5aeb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:51.574 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5aeb7fe640 1 -- 172.21.15.57:0/4151541429 <== mon.0 v1:172.21.15.57:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f5ae4001cb0 con 0x7f5aec0af900 2024-03-29T10:47:51.574 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5af2df4640 5 monclient: authenticate success, global_id 4503 2024-03-29T10:47:51.574 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5af2df4640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:51.574 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.586+0000 7f5af2df4640 4 set_mon_vals no callback set 2024-03-29T10:47:51.574 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.588+0000 7f5af2df4640 10 monclient: shutdown 2024-03-29T10:47:51.575 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.589+0000 7f5aeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.575 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.589+0000 7f5aeb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:51.575 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.589+0000 7f5aeb7fe640 10 monclient: dump: 2024-03-29T10:47:51.575 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:51.575 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:51.575 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:51.575 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:51.575 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.575 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:51.575 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:51.575 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:51.575 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:51.576 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:51.576 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.590+0000 7f5af2df4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:51.576 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.590+0000 7f5af2df4640 1 -- 172.21.15.57:0/4151541429 >> v1:172.21.15.57:6789/0 conn(0x7f5aec0af900 legacy=0x7f5aec13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.576 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.590+0000 7f5af2df4640 1 -- 172.21.15.57:0/4151541429 shutdown_connections 2024-03-29T10:47:51.576 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.590+0000 7f5af2df4640 1 -- 172.21.15.57:0/4151541429 >> 172.21.15.57:0/4151541429 conn(0x7f5aec0b35e0 msgr2=0x7f5aec0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:51.576 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.591+0000 7f5af2df4640 1 -- 172.21.15.57:0/4151541429 shutdown_connections 2024-03-29T10:47:51.577 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.594+0000 7f5af2df4640 1 -- 172.21.15.57:0/4151541429 wait complete. 2024-03-29T10:47:51.579 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.595+0000 7f5af2df4640 10 monclient: build_initial_monmap 2024-03-29T10:47:51.579 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.595+0000 7f5af2df4640 10 monclient: monmap: 2024-03-29T10:47:51.579 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:47:51.579 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:51.579 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:47:51.579 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:47:51.579 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:51.579 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:51.579 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:47:51.579 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:47:51.580 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:47:51.580 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:51.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.595+0000 7f5af2df4640 1 librados: starting msgr at 2024-03-29T10:47:51.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.596+0000 7f5af2df4640 1 librados: starting objecter 2024-03-29T10:47:51.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.596+0000 7f5af2df4640 1 Processor -- start 2024-03-29T10:47:51.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.596+0000 7f5af2df4640 1 -- start start 2024-03-29T10:47:51.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.596+0000 7f5af2df4640 1 librados: setting wanted keys 2024-03-29T10:47:51.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.596+0000 7f5af2df4640 1 librados: calling monclient init 2024-03-29T10:47:51.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.596+0000 7f5af2df4640 10 monclient: init 2024-03-29T10:47:51.581 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.597+0000 7f5af2df4640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:51.581 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.598+0000 7f5af2df4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:47:51.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.598+0000 7f5af2df4640 10 monclient(hunting): picked mon.noname-a con 0x7f5aec0af900 addr v1:172.21.15.57:6789/0 2024-03-29T10:47:51.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.600+0000 7f5af2df4640 10 monclient(hunting): picked mon.noname-c con 0x7f5aec0b4100 addr v1:172.21.15.57:6790/0 2024-03-29T10:47:51.587 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.600+0000 7f5af2df4640 10 monclient(hunting): picked mon.noname-b con 0x7f5aec13ee90 addr v1:172.21.15.179:6789/0 2024-03-29T10:47:51.587 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.604+0000 7f5af1391640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f5aec13ee90 0x7f5aec1e1fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:51333/0 (socket says 172.21.15.57:51333) 2024-03-29T10:47:51.587 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.604+0000 7f5af1391640 1 -- 172.21.15.57:0/2192003316 learned_addr learned my addr 172.21.15.57:0/2192003316 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:47:51.587 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.604+0000 7f5af2df4640 1 -- 172.21.15.57:0/2192003316 --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec070730 con 0x7f5aec0af900 2024-03-29T10:47:51.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.604+0000 7f5af2df4640 1 -- 172.21.15.57:0/2192003316 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec070a10 con 0x7f5aec13ee90 2024-03-29T10:47:51.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.604+0000 7f5af2df4640 1 -- 172.21.15.57:0/2192003316 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec142470 con 0x7f5aec0b4100 2024-03-29T10:47:51.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.604+0000 7f5af2df4640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:51.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.604+0000 7f5af2df4640 10 monclient(hunting): authenticate will time out at 1423.781454s 2024-03-29T10:47:51.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.604+0000 7f5ae97fa640 10 client.?.objecter ms_handle_connect 0x7f5aec13ee90 2024-03-29T10:47:51.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.604+0000 7f5ae97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:51.592 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 10 client.?.objecter ms_handle_connect 0x7f5aec0af900 2024-03-29T10:47:51.592 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:51.592 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 1 -- 172.21.15.57:0/2192003316 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2194417347 0 0) 0x7f5aec070a10 con 0x7f5aec13ee90 2024-03-29T10:47:51.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:51.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:51.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:51.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 10 monclient(hunting): my global_id is 4504 2024-03-29T10:47:51.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 1 -- 172.21.15.57:0/2192003316 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ad0004eb0 con 0x7f5aec13ee90 2024-03-29T10:47:51.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 1 -- 172.21.15.57:0/2192003316 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762222017 0 0) 0x7f5aec070730 con 0x7f5aec0af900 2024-03-29T10:47:51.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:47:51.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:51.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:51.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 10 monclient(hunting): my global_id is 4506 2024-03-29T10:47:51.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 1 -- 172.21.15.57:0/2192003316 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aec070a10 con 0x7f5aec0af900 2024-03-29T10:47:51.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 10 client.?.objecter ms_handle_connect 0x7f5aec0b4100 2024-03-29T10:47:51.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:51.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 1 -- 172.21.15.57:0/2192003316 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1098174724 0 0) 0x7f5ad0004eb0 con 0x7f5aec13ee90 2024-03-29T10:47:51.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 1 -- 172.21.15.57:0/2192003316 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5aec070730 con 0x7f5aec13ee90 2024-03-29T10:47:51.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 1 -- 172.21.15.57:0/2192003316 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f5ae4001550 con 0x7f5aec13ee90 2024-03-29T10:47:51.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:51.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 10 monclient(hunting): dump: 2024-03-29T10:47:51.594 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:51.594 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:51.594 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:51.594 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:51.594 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.594 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:51.594 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:51.594 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:51.594 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:51.594 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:51.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:51.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 1 -- 172.21.15.57:0/2192003316 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2794363508 0 0) 0x7f5aec070a10 con 0x7f5aec0af900 2024-03-29T10:47:51.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 1 -- 172.21.15.57:0/2192003316 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ad0004eb0 con 0x7f5aec0af900 2024-03-29T10:47:51.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 1 -- 172.21.15.57:0/2192003316 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f5ad8001830 con 0x7f5aec0af900 2024-03-29T10:47:51.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:51.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 10 monclient(hunting): dump: 2024-03-29T10:47:51.595 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:51.595 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:51.595 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:51.595 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:51.595 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.595 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:51.595 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:51.595 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:51.596 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:51.596 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:51.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 1 -- 172.21.15.57:0/2192003316 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3648823136 0 0) 0x7f5ad0004eb0 con 0x7f5aec0af900 2024-03-29T10:47:51.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.605+0000 7f5ae97fa640 1 -- 172.21.15.57:0/2192003316 >> v1:172.21.15.57:6790/0 conn(0x7f5aec0b4100 legacy=0x7f5aec06e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.606+0000 7f5ae97fa640 1 -- 172.21.15.57:0/2192003316 >> v1:172.21.15.179:6789/0 conn(0x7f5aec13ee90 legacy=0x7f5aec1e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.606+0000 7f5ae97fa640 10 monclient: _finish_hunting 0 2024-03-29T10:47:51.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.606+0000 7f5ae97fa640 1 monclient: found mon.a 2024-03-29T10:47:51.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.606+0000 7f5ae97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:51.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.606+0000 7f5ae97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:47:51.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.606+0000 7f5ae97fa640 1 -- 172.21.15.57:0/2192003316 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec142600 con 0x7f5aec0af900 2024-03-29T10:47:51.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.606+0000 7f5ae97fa640 10 monclient: _finish_auth 0 2024-03-29T10:47:51.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.606+0000 7f5ae97fa640 10 monclient: _check_auth_tickets 2024-03-29T10:47:51.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.606+0000 7f5ae97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:51.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.606+0000 7f5ae97fa640 1 -- 172.21.15.57:0/2192003316 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ad8001ce0 con 0x7f5aec0af900 2024-03-29T10:47:51.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.606+0000 7f5ae97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:51.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.606+0000 7f5aeaffd640 4 set_mon_vals no callback set 2024-03-29T10:47:51.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.607+0000 7f5ae97fa640 1 -- 172.21.15.57:0/2192003316 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f5ad80037b0 con 0x7f5aec0af900 2024-03-29T10:47:51.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.607+0000 7f5ae97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:51.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.607+0000 7f5ae97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:47:51.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.607+0000 7f5ae97fa640 10 monclient: dump: 2024-03-29T10:47:51.597 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:51.597 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:51.597 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:51.597 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:51.597 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:51.597 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:51.597 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:51.597 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:51.597 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:51.597 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:51.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.608+0000 7f5af2df4640 5 monclient: authenticate success, global_id 4506 2024-03-29T10:47:51.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.608+0000 7f5af2df4640 10 monclient: _renew_subs 2024-03-29T10:47:51.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.608+0000 7f5af2df4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:47:51.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.608+0000 7f5af2df4640 1 -- 172.21.15.57:0/2192003316 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aec06eeb0 con 0x7f5aec0af900 2024-03-29T10:47:51.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.608+0000 7f5af2df4640 10 client.4506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:51.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.608+0000 7f5af2df4640 10 monclient: _renew_subs 2024-03-29T10:47:51.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.608+0000 7f5af2df4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:47:51.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.608+0000 7f5af2df4640 1 -- 172.21.15.57:0/2192003316 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aec0721b0 con 0x7f5aec0af900 2024-03-29T10:47:51.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.608+0000 7f5af2df4640 1 librados: init done 2024-03-29T10:47:51.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.612+0000 7f5af2df4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:51.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.612+0000 7f5af2df4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:51.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.612+0000 7f5af2df4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:47:51.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.612+0000 7f5af2df4640 1 -- 172.21.15.57:0/2192003316 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aec14cda0 con 0x7f5aec0af900 2024-03-29T10:47:51.599 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.613+0000 7f5ae97fa640 1 -- 172.21.15.57:0/2192003316 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f5ad8003910 con 0x7f5aec0af900 2024-03-29T10:47:51.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.613+0000 7f5ae97fa640 1 -- 172.21.15.57:0/2192003316 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f5ad8053630 con 0x7f5aec0af900 2024-03-29T10:47:51.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.613+0000 7f5ae97fa640 10 client.4506.objecter ms_dispatch 0x7f5aec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:51.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.613+0000 7f5ae97fa640 3 client.4506.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:51.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.613+0000 7f5ae97fa640 3 client.4506.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:51.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.613+0000 7f5ae97fa640 20 client.4506.objecter dump_active .. 0 homeless 2024-03-29T10:47:51.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.614+0000 7f5aebfff640 10 monclient: get_auth_request con 0x7f5ad006dc70 auth_method 0 2024-03-29T10:47:51.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.615+0000 7f5ae97fa640 10 client.4506.objecter ms_handle_connect 0x7f5ad006dc70 2024-03-29T10:47:51.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.624+0000 7f5ae97fa640 1 -- 172.21.15.57:0/2192003316 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ad804f8d0 con 0x7f5aec0af900 2024-03-29T10:47:51.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.624+0000 7f5ae97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:51.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.624+0000 7f5ae97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:51.938 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.955+0000 7f5af2df4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:51.938 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.955+0000 7f5af2df4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:51.938 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.955+0000 7f5af2df4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:47:51.938 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.955+0000 7f5af2df4640 1 -- 172.21.15.57:0/2192003316 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5aec06f040 con 0x7f5aec0af900 2024-03-29T10:47:51.943 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.960+0000 7f5ae97fa640 1 -- 172.21.15.57:0/2192003316 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f5ad804f8d0 con 0x7f5aec0af900 2024-03-29T10:47:51.943 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.960+0000 7f5ae97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:51.943 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.960+0000 7f5ae97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:51.943 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:47:51.943 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:51.946 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.963+0000 7f5af2df4640 10 librados: watch_flush enter 2024-03-29T10:47:51.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.965+0000 7f5af2df4640 10 librados: watch_flush exit 2024-03-29T10:47:51.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.965+0000 7f5af2df4640 20 client.4506.objecter shutdown clearing up homeless session... 2024-03-29T10:47:51.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.965+0000 7f5af2df4640 10 client.4506.objecter successfully canceled tick 2024-03-29T10:47:51.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.965+0000 7f5af2df4640 1 -- 172.21.15.57:0/2192003316 >> v1:172.21.15.179:6816/2698295936 conn(0x7f5ad006dc70 legacy=0x7f5ad0070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.966+0000 7f5af2df4640 10 monclient: shutdown 2024-03-29T10:47:51.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.966+0000 7f5af2df4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:51.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.966+0000 7f5af2df4640 1 -- 172.21.15.57:0/2192003316 >> v1:172.21.15.57:6789/0 conn(0x7f5aec0af900 legacy=0x7f5aec13fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:51.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.966+0000 7f5af2df4640 1 -- 172.21.15.57:0/2192003316 shutdown_connections 2024-03-29T10:47:51.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.966+0000 7f5af2df4640 1 -- 172.21.15.57:0/2192003316 >> 172.21.15.57:0/2192003316 conn(0x7f5aec0b35e0 msgr2=0x7f5aec0ab9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:51.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.966+0000 7f5af2df4640 1 -- 172.21.15.57:0/2192003316 shutdown_connections 2024-03-29T10:47:51.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.967+0000 7f5af2df4640 1 -- 172.21.15.57:0/2192003316 wait complete. 2024-03-29T10:47:51.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:51.967+0000 7f5af2df4640 1 librados: shutdown 2024-03-29T10:47:51.967 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:47:58.774 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:47:58.923 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.940+0000 7fba03fff640 10 monclient: get_monmap_and_config 2024-03-29T10:47:58.924 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.940+0000 7fba03fff640 10 monclient: build_initial_monmap 2024-03-29T10:47:58.924 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.940+0000 7fba03fff640 10 monclient: monmap: 2024-03-29T10:47:58.924 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:47:58.924 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:58.924 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:47:58.942136+0000 2024-03-29T10:47:58.924 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:47:58.942136+0000 2024-03-29T10:47:58.924 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:58.924 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:58.924 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:47:58.924 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:47:58.924 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:47:58.924 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:58.925 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.942+0000 7fba03fff640 1 Processor -- start 2024-03-29T10:47:58.925 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.942+0000 7fba03fff640 1 -- start start 2024-03-29T10:47:58.926 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.942+0000 7fba03fff640 10 monclient: init 2024-03-29T10:47:58.926 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.943+0000 7fba03fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:58.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.943+0000 7fba03fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:47:58.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.943+0000 7fba03fff640 10 monclient(hunting): picked mon.noname-a con 0x7fb9fc006eb0 addr v1:172.21.15.57:6789/0 2024-03-29T10:47:58.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.943+0000 7fba03fff640 10 monclient(hunting): picked mon.noname-c con 0x7fb9fc038f10 addr v1:172.21.15.179:6789/0 2024-03-29T10:47:58.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.943+0000 7fba03fff640 10 monclient(hunting): picked mon.noname-b con 0x7fb9fc036690 addr v1:172.21.15.57:6790/0 2024-03-29T10:47:58.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.943+0000 7fba03fff640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9fc033c90 con 0x7fb9fc006eb0 2024-03-29T10:47:58.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.943+0000 7fba03fff640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9fc0389f0 con 0x7fb9fc038f10 2024-03-29T10:47:58.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.943+0000 7fba03fff640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9fc030890 con 0x7fb9fc036690 2024-03-29T10:47:58.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.943+0000 7fba03fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:58.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.943+0000 7fba03fff640 10 monclient(hunting): authenticate will time out at 1431.120515s 2024-03-29T10:47:58.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.946+0000 7fba037fe640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7fb9fc036690 0x7fb9fc036b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:40625/0 (socket says 172.21.15.57:40625) 2024-03-29T10:47:58.932 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.946+0000 7fba037fe640 1 -- 172.21.15.57:0/2613922320 learned_addr learned my addr 172.21.15.57:0/2613922320 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:47:58.938 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 1 -- 172.21.15.57:0/2613922320 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2902782496 0 0) 0x7fb9fc030890 con 0x7fb9fc036690 2024-03-29T10:47:58.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:47:58.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:58.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:58.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 10 monclient(hunting): my global_id is 4553 2024-03-29T10:47:58.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 1 -- 172.21.15.57:0/2613922320 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9ec003520 con 0x7fb9fc036690 2024-03-29T10:47:58.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 1 -- 172.21.15.57:0/2613922320 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2912784544 0 0) 0x7fb9ec003520 con 0x7fb9fc036690 2024-03-29T10:47:58.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 1 -- 172.21.15.57:0/2613922320 >> v1:172.21.15.179:6789/0 conn(0x7fb9fc038f10 legacy=0x7fb9fc039330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:47:58.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 1 -- 172.21.15.57:0/2613922320 >> v1:172.21.15.57:6789/0 conn(0x7fb9fc006eb0 legacy=0x7fb9fc031760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:58.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:47:58.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 1 monclient: found mon.noname-b 2024-03-29T10:47:58.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:58.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:47:58.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 1 -- 172.21.15.57:0/2613922320 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9fc0cbc40 con 0x7fb9fc036690 2024-03-29T10:47:58.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 10 monclient: _finish_auth 0 2024-03-29T10:47:58.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:47:58.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:58.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 1 -- 172.21.15.57:0/2613922320 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fba0404f730 con 0x7fb9fc036690 2024-03-29T10:47:58.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:58.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:47:58.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 10 monclient: dump: 2024-03-29T10:47:58.940 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:58.940 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:58.941 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:58.941 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:58.941 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:58.941 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:58.941 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:58.941 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:58.941 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:58.941 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:58.941 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 1 -- 172.21.15.57:0/2613922320 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba04075f10 con 0x7fb9fc036690 2024-03-29T10:47:58.941 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:58.941 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 1 -- 172.21.15.57:0/2613922320 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fba040758b0 con 0x7fb9fc036690 2024-03-29T10:47:58.941 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:58.942 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:58.942 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.951+0000 7fba01ffb640 10 monclient: dump: 2024-03-29T10:47:58.942 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:58.942 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:58.942 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:58.942 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:58.942 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:58.942 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:58.942 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:58.942 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:58.942 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:58.942 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:58.942 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.953+0000 7fba03fff640 5 monclient: authenticate success, global_id 4553 2024-03-29T10:47:58.943 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.953+0000 7fba03fff640 10 monclient: get_monmap_and_config success 2024-03-29T10:47:58.943 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.953+0000 7fba03fff640 4 set_mon_vals no callback set 2024-03-29T10:47:58.943 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.954+0000 7fba03fff640 10 monclient: shutdown 2024-03-29T10:47:58.943 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.954+0000 7fba03fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:58.943 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.954+0000 7fba03fff640 1 -- 172.21.15.57:0/2613922320 >> v1:172.21.15.57:6790/0 conn(0x7fb9fc036690 legacy=0x7fb9fc036b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:58.943 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.954+0000 7fba03fff640 1 -- 172.21.15.57:0/2613922320 shutdown_connections 2024-03-29T10:47:58.943 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.954+0000 7fba03fff640 1 -- 172.21.15.57:0/2613922320 >> 172.21.15.57:0/2613922320 conn(0x7fb9fc005c20 msgr2=0x7fb9fc006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:58.943 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.956+0000 7fba03fff640 1 -- 172.21.15.57:0/2613922320 shutdown_connections 2024-03-29T10:47:58.945 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.956+0000 7fba03fff640 1 -- 172.21.15.57:0/2613922320 wait complete. 2024-03-29T10:47:58.945 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.958+0000 7fba03fff640 10 monclient: build_initial_monmap 2024-03-29T10:47:58.945 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.958+0000 7fba03fff640 10 monclient: monmap: 2024-03-29T10:47:58.945 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:47:58.945 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:47:58.945 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:47:58.945 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:47:58.945 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:47:58.945 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:58.945 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:47:58.945 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:47:58.945 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:47:58.946 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:58.946 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.958+0000 7fba03fff640 1 librados: starting msgr at 2024-03-29T10:47:58.946 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.958+0000 7fba03fff640 1 librados: starting objecter 2024-03-29T10:47:58.946 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.958+0000 7fba03fff640 1 Processor -- start 2024-03-29T10:47:58.946 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.959+0000 7fba03fff640 1 -- start start 2024-03-29T10:47:58.946 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.959+0000 7fba03fff640 1 librados: setting wanted keys 2024-03-29T10:47:58.946 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.959+0000 7fba03fff640 1 librados: calling monclient init 2024-03-29T10:47:58.946 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.959+0000 7fba03fff640 10 monclient: init 2024-03-29T10:47:58.946 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.959+0000 7fba03fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:47:58.946 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.959+0000 7fba03fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:47:58.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.962+0000 7fba03fff640 10 monclient(hunting): picked mon.noname-c con 0x7fb9fc006eb0 addr v1:172.21.15.57:6790/0 2024-03-29T10:47:58.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.962+0000 7fba03fff640 10 monclient(hunting): picked mon.noname-a con 0x7fb9fc036690 addr v1:172.21.15.57:6789/0 2024-03-29T10:47:58.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.962+0000 7fba03fff640 10 monclient(hunting): picked mon.noname-b con 0x7fb9fc038f10 addr v1:172.21.15.179:6789/0 2024-03-29T10:47:58.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.962+0000 7fba03fff640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9fc0cae60 con 0x7fb9fc036690 2024-03-29T10:47:58.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.962+0000 7fba03fff640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9fc02ea70 con 0x7fb9fc038f10 2024-03-29T10:47:58.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.962+0000 7fba03fff640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9fc02ed50 con 0x7fb9fc006eb0 2024-03-29T10:47:58.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.962+0000 7fba03fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:47:58.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.962+0000 7fba03fff640 10 monclient(hunting): authenticate will time out at 1431.139704s 2024-03-29T10:47:58.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.962+0000 7fba02ffd640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7fb9fc006eb0 0x7fb9fc02f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:40699/0 (socket says 172.21.15.57:40699) 2024-03-29T10:47:58.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.962+0000 7fba02ffd640 1 -- 172.21.15.57:0/1137248884 learned_addr learned my addr 172.21.15.57:0/1137248884 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:47:58.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.963+0000 7fb9eb7fe640 10 client.?.objecter ms_handle_connect 0x7fb9fc006eb0 2024-03-29T10:47:58.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.963+0000 7fb9eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:58.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.963+0000 7fb9eb7fe640 1 -- 172.21.15.57:0/1137248884 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828141099 0 0) 0x7fb9fc02ed50 con 0x7fb9fc006eb0 2024-03-29T10:47:58.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.963+0000 7fb9eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:58.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.963+0000 7fb9eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:58.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.963+0000 7fb9eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:58.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.963+0000 7fb9eb7fe640 10 monclient(hunting): my global_id is 4556 2024-03-29T10:47:58.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.963+0000 7fb9eb7fe640 1 -- 172.21.15.57:0/1137248884 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9f0004eb0 con 0x7fb9fc006eb0 2024-03-29T10:47:58.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.964+0000 7fb9eb7fe640 1 -- 172.21.15.57:0/1137248884 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1569572298 0 0) 0x7fb9f0004eb0 con 0x7fb9fc006eb0 2024-03-29T10:47:58.951 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.965+0000 7fb9eb7fe640 1 -- 172.21.15.57:0/1137248884 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9fc02ed50 con 0x7fb9fc006eb0 2024-03-29T10:47:58.951 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.965+0000 7fb9eb7fe640 1 -- 172.21.15.57:0/1137248884 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb9f4001a10 con 0x7fb9fc006eb0 2024-03-29T10:47:58.951 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.965+0000 7fb9eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:58.951 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.965+0000 7fb9eb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:47:58.951 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.965+0000 7fb9eb7fe640 10 monclient(hunting): dump: 2024-03-29T10:47:58.951 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:58.951 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:58.951 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:58.951 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:58.951 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:58.952 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:58.952 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:58.952 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:58.952 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:58.952 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:58.952 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.965+0000 7fb9eb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:47:58.952 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.966+0000 7fb9eb7fe640 10 client.?.objecter ms_handle_connect 0x7fb9fc038f10 2024-03-29T10:47:58.952 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.966+0000 7fb9eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:47:58.952 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.966+0000 7fb9eb7fe640 1 -- 172.21.15.57:0/1137248884 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177668231 0 0) 0x7fb9fc02ea70 con 0x7fb9fc038f10 2024-03-29T10:47:58.952 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.966+0000 7fb9eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:47:58.952 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.966+0000 7fb9eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:47:58.952 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.966+0000 7fb9eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:47:58.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.966+0000 7fb9eb7fe640 10 monclient(hunting): my global_id is 4507 2024-03-29T10:47:58.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.966+0000 7fb9eb7fe640 1 -- 172.21.15.57:0/1137248884 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9f0004eb0 con 0x7fb9fc038f10 2024-03-29T10:47:58.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.966+0000 7fb9eb7fe640 1 -- 172.21.15.57:0/1137248884 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 832115621 0 0) 0x7fb9fc02ed50 con 0x7fb9fc006eb0 2024-03-29T10:47:58.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.966+0000 7fb9eb7fe640 1 -- 172.21.15.57:0/1137248884 >> v1:172.21.15.179:6789/0 conn(0x7fb9fc038f10 legacy=0x7fb9fc16c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:58.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.966+0000 7fb9eb7fe640 1 -- 172.21.15.57:0/1137248884 >> v1:172.21.15.57:6789/0 conn(0x7fb9fc036690 legacy=0x7fb9fc168960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:47:58.955 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.971+0000 7fb9eb7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:47:58.955 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.971+0000 7fb9eb7fe640 1 monclient: found mon.c 2024-03-29T10:47:58.955 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.971+0000 7fb9eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:47:58.955 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.971+0000 7fb9eb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:47:58.955 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.971+0000 7fb9eb7fe640 1 -- 172.21.15.57:0/1137248884 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9fc02f030 con 0x7fb9fc006eb0 2024-03-29T10:47:58.955 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.971+0000 7fb9eb7fe640 10 monclient: _finish_auth 0 2024-03-29T10:47:58.955 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.971+0000 7fb9eb7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:47:58.955 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.971+0000 7fb9eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:47:58.955 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.971+0000 7fba03fff640 5 monclient: authenticate success, global_id 4556 2024-03-29T10:47:58.955 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.971+0000 7fba03fff640 10 monclient: _renew_subs 2024-03-29T10:47:58.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.971+0000 7fba03fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:47:58.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.971+0000 7fba03fff640 1 -- 172.21.15.57:0/1137248884 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9fc0cb340 con 0x7fb9fc006eb0 2024-03-29T10:47:58.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.971+0000 7fba03fff640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:47:58.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.971+0000 7fba03fff640 10 monclient: _renew_subs 2024-03-29T10:47:58.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.971+0000 7fba03fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:47:58.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.971+0000 7fba03fff640 1 -- 172.21.15.57:0/1137248884 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9fc02d350 con 0x7fb9fc006eb0 2024-03-29T10:47:58.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.971+0000 7fba03fff640 1 librados: init done 2024-03-29T10:47:58.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.972+0000 7fb9eb7fe640 1 -- 172.21.15.57:0/1137248884 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9f4001d10 con 0x7fb9fc006eb0 2024-03-29T10:47:58.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.972+0000 7fb9eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:47:58.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.972+0000 7fba017fa640 4 set_mon_vals no callback set 2024-03-29T10:47:58.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.973+0000 7fb9eb7fe640 1 -- 172.21.15.57:0/1137248884 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb9f4003a60 con 0x7fb9fc006eb0 2024-03-29T10:47:58.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.973+0000 7fb9eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:47:58.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.973+0000 7fb9eb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:47:58.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.973+0000 7fb9eb7fe640 10 monclient: dump: 2024-03-29T10:47:58.957 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:47:58.957 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:47:58.957 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:58.957 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:47:58.957 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:47:58.958 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:47:58.958 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:47:58.958 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:47:58.958 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:47:58.958 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:47:58.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.974+0000 7fba03fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:58.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.974+0000 7fba03fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:58.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.974+0000 7fba03fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:47:58.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.974+0000 7fba03fff640 1 -- 172.21.15.57:0/1137248884 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e4005080 con 0x7fb9fc006eb0 2024-03-29T10:47:58.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.976+0000 7fb9eb7fe640 1 -- 172.21.15.57:0/1137248884 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fb9f4003bc0 con 0x7fb9fc006eb0 2024-03-29T10:47:58.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.977+0000 7fb9eb7fe640 1 -- 172.21.15.57:0/1137248884 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7fb9f4084c00 con 0x7fb9fc006eb0 2024-03-29T10:47:58.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.977+0000 7fba027fc640 10 monclient: get_auth_request con 0x7fb9f006dc70 auth_method 0 2024-03-29T10:47:58.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.977+0000 7fb9eb7fe640 10 client.4556.objecter ms_dispatch 0x7fba04000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:47:58.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.978+0000 7fb9eb7fe640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:47:58.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.978+0000 7fb9eb7fe640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:47:58.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.978+0000 7fb9eb7fe640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-29T10:47:58.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.978+0000 7fb9eb7fe640 10 client.4556.objecter ms_handle_connect 0x7fb9f006dc70 2024-03-29T10:47:58.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.982+0000 7fb9eb7fe640 1 -- 172.21.15.57:0/1137248884 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb9f4050310 con 0x7fb9fc006eb0 2024-03-29T10:47:58.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.983+0000 7fb9eb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:47:58.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:58.983+0000 7fb9eb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:47:59.346 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:59.362+0000 7fba03fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:59.347 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:59.362+0000 7fba03fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:59.347 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:59.362+0000 7fba03fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:47:59.347 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:59.362+0000 7fba03fff640 1 -- 172.21.15.57:0/1137248884 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9e4005330 con 0x7fb9fc006eb0 2024-03-29T10:47:59.347 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:59.363+0000 7fb9eb7fe640 1 -- 172.21.15.57:0/1137248884 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7fb9f4052410 con 0x7fb9fc006eb0 2024-03-29T10:47:59.347 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:59.363+0000 7fb9eb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:47:59.347 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:59.363+0000 7fb9eb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:47:59.347 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:47:59.347 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:47:59.350 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:59.367+0000 7fba03fff640 10 librados: watch_flush enter 2024-03-29T10:47:59.352 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:59.369+0000 7fba03fff640 10 librados: watch_flush exit 2024-03-29T10:47:59.352 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:59.369+0000 7fba03fff640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-29T10:47:59.352 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:59.369+0000 7fba03fff640 10 client.4556.objecter successfully canceled tick 2024-03-29T10:47:59.352 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:59.369+0000 7fba03fff640 1 -- 172.21.15.57:0/1137248884 >> v1:172.21.15.179:6816/2698295936 conn(0x7fb9f006dc70 legacy=0x7fb9f0070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:59.352 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:59.369+0000 7fba03fff640 10 monclient: shutdown 2024-03-29T10:47:59.352 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:59.369+0000 7fba03fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:47:59.352 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:59.369+0000 7fba03fff640 1 -- 172.21.15.57:0/1137248884 >> v1:172.21.15.57:6790/0 conn(0x7fb9fc006eb0 legacy=0x7fb9fc02f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:47:59.353 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:59.370+0000 7fba03fff640 1 -- 172.21.15.57:0/1137248884 shutdown_connections 2024-03-29T10:47:59.353 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:59.370+0000 7fba03fff640 1 -- 172.21.15.57:0/1137248884 >> 172.21.15.57:0/1137248884 conn(0x7fb9fc005c20 msgr2=0x7fb9fc0050f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:47:59.353 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:59.370+0000 7fba03fff640 1 -- 172.21.15.57:0/1137248884 shutdown_connections 2024-03-29T10:47:59.353 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:59.370+0000 7fba03fff640 1 -- 172.21.15.57:0/1137248884 wait complete. 2024-03-29T10:47:59.354 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:47:59.371+0000 7fba03fff640 1 librados: shutdown 2024-03-29T10:47:59.377 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:06.185 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:06.336 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.351+0000 7f1a8131c640 10 monclient: get_monmap_and_config 2024-03-29T10:48:06.336 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.351+0000 7f1a8131c640 10 monclient: build_initial_monmap 2024-03-29T10:48:06.336 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.351+0000 7f1a8131c640 10 monclient: monmap: 2024-03-29T10:48:06.336 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:48:06.336 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:06.336 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:48:06.353641+0000 2024-03-29T10:48:06.336 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:48:06.353641+0000 2024-03-29T10:48:06.336 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:06.336 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:06.336 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:48:06.336 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:48:06.336 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:48:06.337 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:06.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.353+0000 7f1a8131c640 1 Processor -- start 2024-03-29T10:48:06.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.353+0000 7f1a8131c640 1 -- start start 2024-03-29T10:48:06.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.353+0000 7f1a8131c640 10 monclient: init 2024-03-29T10:48:06.338 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.354+0000 7f1a8131c640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:06.338 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.354+0000 7f1a8131c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:48:06.338 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.354+0000 7f1a8131c640 10 monclient(hunting): picked mon.noname-b con 0x7f1a7c0a9ac0 addr v1:172.21.15.57:6790/0 2024-03-29T10:48:06.338 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.354+0000 7f1a8131c640 10 monclient(hunting): picked mon.noname-a con 0x7f1a7c13ee90 addr v1:172.21.15.57:6789/0 2024-03-29T10:48:06.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.354+0000 7f1a8131c640 10 monclient(hunting): picked mon.noname-c con 0x7f1a7c0af900 addr v1:172.21.15.179:6789/0 2024-03-29T10:48:06.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.354+0000 7f1a8131c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7c0afd10 con 0x7f1a7c13ee90 2024-03-29T10:48:06.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.354+0000 7f1a8131c640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7c0af6f0 con 0x7f1a7c0af900 2024-03-29T10:48:06.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.354+0000 7f1a8131c640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7c0aac80 con 0x7f1a7c0a9ac0 2024-03-29T10:48:06.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.354+0000 7f1a8131c640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:06.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.354+0000 7f1a8131c640 10 monclient(hunting): authenticate will time out at 1438.531281s 2024-03-29T10:48:06.342 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.358+0000 7f1a7a7fc640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1a7c13ee90 0x7f1a7c13b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:37550/0 (socket says 172.21.15.57:37550) 2024-03-29T10:48:06.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.358+0000 7f1a7a7fc640 1 -- 172.21.15.57:0/358632438 learned_addr learned my addr 172.21.15.57:0/358632438 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:48:06.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.358+0000 7f1a79ffb640 1 -- 172.21.15.57:0/358632438 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2109955814 0 0) 0x7f1a7c0afd10 con 0x7f1a7c13ee90 2024-03-29T10:48:06.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.358+0000 7f1a79ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:48:06.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.358+0000 7f1a79ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:06.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.358+0000 7f1a79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:06.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.358+0000 7f1a79ffb640 10 monclient(hunting): my global_id is 4512 2024-03-29T10:48:06.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.358+0000 7f1a79ffb640 1 -- 172.21.15.57:0/358632438 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a64003520 con 0x7f1a7c13ee90 2024-03-29T10:48:06.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a79ffb640 1 -- 172.21.15.57:0/358632438 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1070257719 0 0) 0x7f1a64003520 con 0x7f1a7c13ee90 2024-03-29T10:48:06.343 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a79ffb640 1 -- 172.21.15.57:0/358632438 >> v1:172.21.15.57:6790/0 conn(0x7f1a7c0a9ac0 legacy=0x7f1a7c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:06.344 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a79ffb640 1 -- 172.21.15.57:0/358632438 >> v1:172.21.15.179:6789/0 conn(0x7f1a7c0af900 legacy=0x7f1a7c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:06.344 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a79ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:48:06.344 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a79ffb640 1 monclient: found mon.noname-a 2024-03-29T10:48:06.344 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a79ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:06.344 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a79ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:48:06.347 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a79ffb640 1 -- 172.21.15.57:0/358632438 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7c0aaf60 con 0x7f1a7c13ee90 2024-03-29T10:48:06.347 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a79ffb640 10 monclient: _finish_auth 0 2024-03-29T10:48:06.347 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a79ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:48:06.348 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:06.348 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a79ffb640 1 -- 172.21.15.57:0/358632438 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1a74001a10 con 0x7f1a7c13ee90 2024-03-29T10:48:06.348 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:06.348 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a79ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:06.348 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a79ffb640 10 monclient: dump: 2024-03-29T10:48:06.348 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:06.348 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:06.348 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:06.348 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:06.348 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:06.348 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:06.348 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:06.348 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:06.348 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:06.348 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:06.349 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a8131c640 5 monclient: authenticate success, global_id 4512 2024-03-29T10:48:06.349 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a8131c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:06.349 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a79ffb640 1 -- 172.21.15.57:0/358632438 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a74001b70 con 0x7f1a7c13ee90 2024-03-29T10:48:06.349 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a79ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:06.349 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a79ffb640 1 -- 172.21.15.57:0/358632438 <== mon.0 v1:172.21.15.57:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1a74001fc0 con 0x7f1a7c13ee90 2024-03-29T10:48:06.349 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:06.349 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a79ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:06.349 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a79ffb640 10 monclient: dump: 2024-03-29T10:48:06.349 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:06.349 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:06.349 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:06.349 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:06.349 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:06.349 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:06.349 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:06.350 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:06.350 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:06.350 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:06.350 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a8131c640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:06.350 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.359+0000 7f1a8131c640 4 set_mon_vals no callback set 2024-03-29T10:48:06.350 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.360+0000 7f1a8131c640 10 monclient: shutdown 2024-03-29T10:48:06.350 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.360+0000 7f1a8131c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:06.350 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.360+0000 7f1a8131c640 1 -- 172.21.15.57:0/358632438 >> v1:172.21.15.57:6789/0 conn(0x7f1a7c13ee90 legacy=0x7f1a7c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:06.350 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.362+0000 7f1a8131c640 1 -- 172.21.15.57:0/358632438 shutdown_connections 2024-03-29T10:48:06.350 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.362+0000 7f1a8131c640 1 -- 172.21.15.57:0/358632438 >> 172.21.15.57:0/358632438 conn(0x7f1a7c0b35e0 msgr2=0x7f1a7c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:06.350 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.362+0000 7f1a8131c640 1 -- 172.21.15.57:0/358632438 shutdown_connections 2024-03-29T10:48:06.350 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.362+0000 7f1a8131c640 1 -- 172.21.15.57:0/358632438 wait complete. 2024-03-29T10:48:06.350 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.363+0000 7f1a8131c640 10 monclient: build_initial_monmap 2024-03-29T10:48:06.350 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.363+0000 7f1a8131c640 10 monclient: monmap: 2024-03-29T10:48:06.350 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:48:06.351 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:06.351 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:48:06.351 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:48:06.351 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:06.351 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:06.351 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:48:06.351 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:48:06.351 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:48:06.351 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:06.351 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.364+0000 7f1a8131c640 1 librados: starting msgr at 2024-03-29T10:48:06.351 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.364+0000 7f1a8131c640 1 librados: starting objecter 2024-03-29T10:48:06.351 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.364+0000 7f1a8131c640 1 Processor -- start 2024-03-29T10:48:06.351 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.364+0000 7f1a8131c640 1 -- start start 2024-03-29T10:48:06.351 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.364+0000 7f1a8131c640 1 librados: setting wanted keys 2024-03-29T10:48:06.351 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.364+0000 7f1a8131c640 1 librados: calling monclient init 2024-03-29T10:48:06.352 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.364+0000 7f1a8131c640 10 monclient: init 2024-03-29T10:48:06.355 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.365+0000 7f1a8131c640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:06.355 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.365+0000 7f1a8131c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:48:06.356 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.365+0000 7f1a8131c640 10 monclient(hunting): picked mon.noname-a con 0x7f1a7c0a9ac0 addr v1:172.21.15.57:6789/0 2024-03-29T10:48:06.356 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.365+0000 7f1a8131c640 10 monclient(hunting): picked mon.noname-c con 0x7f1a7c0af900 addr v1:172.21.15.57:6790/0 2024-03-29T10:48:06.356 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.365+0000 7f1a8131c640 10 monclient(hunting): picked mon.noname-b con 0x7f1a7c13ee90 addr v1:172.21.15.179:6789/0 2024-03-29T10:48:06.356 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.365+0000 7f1a8131c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7c1db430 con 0x7f1a7c0a9ac0 2024-03-29T10:48:06.356 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.365+0000 7f1a8131c640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7c06fe30 con 0x7f1a7c13ee90 2024-03-29T10:48:06.356 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.365+0000 7f1a8131c640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7c06ffc0 con 0x7f1a7c0af900 2024-03-29T10:48:06.356 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.365+0000 7f1a8131c640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:06.356 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.365+0000 7f1a8131c640 10 monclient(hunting): authenticate will time out at 1438.542747s 2024-03-29T10:48:06.356 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.368+0000 7f1a7affd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1a7c0a9ac0 0x7f1a7c1db950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:38254/0 (socket says 172.21.15.57:38254) 2024-03-29T10:48:06.356 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.369+0000 7f1a7affd640 1 -- 172.21.15.57:0/662556822 learned_addr learned my addr 172.21.15.57:0/662556822 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:48:06.356 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.370+0000 7f1a637fe640 10 client.?.objecter ms_handle_connect 0x7f1a7c0a9ac0 2024-03-29T10:48:06.356 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.370+0000 7f1a637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:06.356 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.370+0000 7f1a637fe640 10 client.?.objecter ms_handle_connect 0x7f1a7c13ee90 2024-03-29T10:48:06.357 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.370+0000 7f1a637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:06.357 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.370+0000 7f1a637fe640 1 -- 172.21.15.57:0/662556822 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 456039565 0 0) 0x7f1a7c06fe30 con 0x7f1a7c13ee90 2024-03-29T10:48:06.357 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.370+0000 7f1a637fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:06.357 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.370+0000 7f1a637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:06.357 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.370+0000 7f1a637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:06.357 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.370+0000 7f1a637fe640 10 monclient(hunting): my global_id is 4513 2024-03-29T10:48:06.357 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.370+0000 7f1a637fe640 1 -- 172.21.15.57:0/662556822 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a68004eb0 con 0x7f1a7c13ee90 2024-03-29T10:48:06.357 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.370+0000 7f1a637fe640 1 -- 172.21.15.57:0/662556822 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093506560 0 0) 0x7f1a7c1db430 con 0x7f1a7c0a9ac0 2024-03-29T10:48:06.357 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.370+0000 7f1a637fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:06.357 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.370+0000 7f1a637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:06.357 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.370+0000 7f1a637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:06.357 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.370+0000 7f1a637fe640 10 monclient(hunting): my global_id is 4515 2024-03-29T10:48:06.357 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.370+0000 7f1a637fe640 1 -- 172.21.15.57:0/662556822 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a7c06fe30 con 0x7f1a7c0a9ac0 2024-03-29T10:48:06.357 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.371+0000 7f1a637fe640 1 -- 172.21.15.57:0/662556822 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 449238405 0 0) 0x7f1a68004eb0 con 0x7f1a7c13ee90 2024-03-29T10:48:06.357 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.371+0000 7f1a637fe640 1 -- 172.21.15.57:0/662556822 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a7c1db430 con 0x7f1a7c13ee90 2024-03-29T10:48:06.358 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.371+0000 7f1a637fe640 1 -- 172.21.15.57:0/662556822 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1a6c001830 con 0x7f1a7c13ee90 2024-03-29T10:48:06.358 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.371+0000 7f1a637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:06.358 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.371+0000 7f1a637fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:06.358 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.371+0000 7f1a637fe640 10 monclient(hunting): dump: 2024-03-29T10:48:06.358 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:06.358 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:06.358 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:06.358 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:06.358 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:06.358 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:06.358 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:06.358 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:06.358 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:06.358 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:06.359 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.371+0000 7f1a637fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:06.359 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 10 client.?.objecter ms_handle_connect 0x7f1a7c0af900 2024-03-29T10:48:06.359 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:06.359 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 1 -- 172.21.15.57:0/662556822 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322082777 0 0) 0x7f1a7c06ffc0 con 0x7f1a7c0af900 2024-03-29T10:48:06.359 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:06.359 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:06.359 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:06.359 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 10 monclient(hunting): my global_id is 4562 2024-03-29T10:48:06.359 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 1 -- 172.21.15.57:0/662556822 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a68005890 con 0x7f1a7c0af900 2024-03-29T10:48:06.359 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 1 -- 172.21.15.57:0/662556822 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 500804297 0 0) 0x7f1a7c06fe30 con 0x7f1a7c0a9ac0 2024-03-29T10:48:06.359 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 1 -- 172.21.15.57:0/662556822 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a68005d20 con 0x7f1a7c0a9ac0 2024-03-29T10:48:06.359 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 1 -- 172.21.15.57:0/662556822 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1a70001950 con 0x7f1a7c0a9ac0 2024-03-29T10:48:06.359 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:06.359 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:06.359 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 10 monclient(hunting): dump: 2024-03-29T10:48:06.360 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:06.360 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:06.360 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:06.360 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:06.360 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:06.360 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:06.360 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:06.360 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:06.360 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:06.360 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:06.360 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 1 -- 172.21.15.57:0/662556822 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2310815236 0 0) 0x7f1a7c1db430 con 0x7f1a7c13ee90 2024-03-29T10:48:06.360 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 1 -- 172.21.15.57:0/662556822 >> v1:172.21.15.57:6790/0 conn(0x7f1a7c0af900 legacy=0x7f1a7c06f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:06.360 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 1 -- 172.21.15.57:0/662556822 >> v1:172.21.15.57:6789/0 conn(0x7f1a7c0a9ac0 legacy=0x7f1a7c1db950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:06.360 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 10 monclient: _finish_hunting 0 2024-03-29T10:48:06.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 1 monclient: found mon.b 2024-03-29T10:48:06.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:06.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:48:06.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 1 -- 172.21.15.57:0/662556822 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7c071580 con 0x7f1a7c13ee90 2024-03-29T10:48:06.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 10 monclient: _finish_auth 0 2024-03-29T10:48:06.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 10 monclient: _check_auth_tickets 2024-03-29T10:48:06.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:06.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a8131c640 5 monclient: authenticate success, global_id 4513 2024-03-29T10:48:06.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a8131c640 10 monclient: _renew_subs 2024-03-29T10:48:06.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a8131c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:48:06.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a8131c640 1 -- 172.21.15.57:0/662556822 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a7c1dc060 con 0x7f1a7c13ee90 2024-03-29T10:48:06.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a8131c640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:06.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a8131c640 10 monclient: _renew_subs 2024-03-29T10:48:06.361 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a8131c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:48:06.362 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a8131c640 1 -- 172.21.15.57:0/662556822 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a7c06faf0 con 0x7f1a7c13ee90 2024-03-29T10:48:06.362 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a8131c640 1 librados: init done 2024-03-29T10:48:06.362 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 1 -- 172.21.15.57:0/662556822 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a6c001d20 con 0x7f1a7c13ee90 2024-03-29T10:48:06.362 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.372+0000 7f1a637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:06.362 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.373+0000 7f1a637fe640 1 -- 172.21.15.57:0/662556822 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1a6c001830 con 0x7f1a7c13ee90 2024-03-29T10:48:06.362 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.373+0000 7f1a637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:06.362 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.373+0000 7f1a637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:06.362 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.373+0000 7f1a637fe640 10 monclient: dump: 2024-03-29T10:48:06.362 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:06.362 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:06.362 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:06.362 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:06.362 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:06.362 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:06.362 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:06.363 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:06.363 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:06.363 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:06.364 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.373+0000 7f1a78ff9640 4 set_mon_vals no callback set 2024-03-29T10:48:06.364 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.375+0000 7f1a8131c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:06.364 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.375+0000 7f1a8131c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:06.364 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.375+0000 7f1a8131c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:48:06.364 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.375+0000 7f1a8131c640 1 -- 172.21.15.57:0/662556822 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5c005080 con 0x7f1a7c13ee90 2024-03-29T10:48:06.364 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.375+0000 7f1a637fe640 1 -- 172.21.15.57:0/662556822 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f1a6c0035f0 con 0x7f1a7c13ee90 2024-03-29T10:48:06.365 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.376+0000 7f1a7affd640 10 monclient: get_auth_request con 0x7f1a6806e6d0 auth_method 0 2024-03-29T10:48:06.365 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.376+0000 7f1a637fe640 1 -- 172.21.15.57:0/662556822 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f1a6c053130 con 0x7f1a7c13ee90 2024-03-29T10:48:06.365 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.376+0000 7f1a637fe640 10 client.4513.objecter ms_dispatch 0x7f1a7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:06.365 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.376+0000 7f1a637fe640 3 client.4513.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:06.365 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.376+0000 7f1a637fe640 3 client.4513.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:06.365 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.376+0000 7f1a637fe640 20 client.4513.objecter dump_active .. 0 homeless 2024-03-29T10:48:06.365 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.376+0000 7f1a637fe640 10 client.4513.objecter ms_handle_connect 0x7f1a6806e6d0 2024-03-29T10:48:06.367 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.384+0000 7f1a637fe640 1 -- 172.21.15.57:0/662556822 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1a6c04f3d0 con 0x7f1a7c13ee90 2024-03-29T10:48:06.367 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.384+0000 7f1a637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:06.367 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.384+0000 7f1a637fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:06.668 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.685+0000 7f1a8131c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:06.668 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.685+0000 7f1a8131c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:06.668 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.685+0000 7f1a8131c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:48:06.668 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.685+0000 7f1a8131c640 1 -- 172.21.15.57:0/662556822 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a5c005330 con 0x7f1a7c13ee90 2024-03-29T10:48:06.669 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.686+0000 7f1a637fe640 1 -- 172.21.15.57:0/662556822 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f1a6c04f3d0 con 0x7f1a7c13ee90 2024-03-29T10:48:06.669 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.686+0000 7f1a637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:06.670 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.686+0000 7f1a637fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:06.670 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:48:06.670 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:06.673 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.690+0000 7f1a8131c640 10 librados: watch_flush enter 2024-03-29T10:48:06.674 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.690+0000 7f1a8131c640 10 librados: watch_flush exit 2024-03-29T10:48:06.674 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.690+0000 7f1a8131c640 20 client.4513.objecter shutdown clearing up homeless session... 2024-03-29T10:48:06.674 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.690+0000 7f1a8131c640 10 client.4513.objecter successfully canceled tick 2024-03-29T10:48:06.674 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.690+0000 7f1a8131c640 1 -- 172.21.15.57:0/662556822 >> v1:172.21.15.179:6816/2698295936 conn(0x7f1a6806e6d0 legacy=0x7f1a68070b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:06.674 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.690+0000 7f1a8131c640 10 monclient: shutdown 2024-03-29T10:48:06.674 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.690+0000 7f1a8131c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:06.674 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.690+0000 7f1a8131c640 1 -- 172.21.15.57:0/662556822 >> v1:172.21.15.179:6789/0 conn(0x7f1a7c13ee90 legacy=0x7f1a7c071be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:06.674 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.690+0000 7f1a8131c640 1 -- 172.21.15.57:0/662556822 shutdown_connections 2024-03-29T10:48:06.674 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.690+0000 7f1a8131c640 1 -- 172.21.15.57:0/662556822 >> 172.21.15.57:0/662556822 conn(0x7f1a7c0b35e0 msgr2=0x7f1a7c1c8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:06.675 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.691+0000 7f1a8131c640 1 -- 172.21.15.57:0/662556822 shutdown_connections 2024-03-29T10:48:06.675 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.691+0000 7f1a8131c640 1 -- 172.21.15.57:0/662556822 wait complete. 2024-03-29T10:48:06.675 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:06.691+0000 7f1a8131c640 1 librados: shutdown 2024-03-29T10:48:06.693 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:13.500 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:13.649 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.666+0000 7f7df2f47640 10 monclient: get_monmap_and_config 2024-03-29T10:48:13.649 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.666+0000 7f7df2f47640 10 monclient: build_initial_monmap 2024-03-29T10:48:13.650 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.666+0000 7f7df2f47640 10 monclient: monmap: 2024-03-29T10:48:13.650 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:48:13.650 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:13.650 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:48:13.668290+0000 2024-03-29T10:48:13.650 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:48:13.668290+0000 2024-03-29T10:48:13.650 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:13.650 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:13.650 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:48:13.650 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:48:13.650 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:48:13.650 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:13.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.667+0000 7f7df2f47640 1 Processor -- start 2024-03-29T10:48:13.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.667+0000 7f7df2f47640 1 -- start start 2024-03-29T10:48:13.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.667+0000 7f7df2f47640 10 monclient: init 2024-03-29T10:48:13.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.668+0000 7f7df2f47640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:13.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.668+0000 7f7df2f47640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:48:13.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.668+0000 7f7df2f47640 10 monclient(hunting): picked mon.noname-a con 0x7f7dec1422b0 addr v1:172.21.15.57:6789/0 2024-03-29T10:48:13.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.668+0000 7f7df2f47640 10 monclient(hunting): picked mon.noname-b con 0x7f7dec142690 addr v1:172.21.15.57:6790/0 2024-03-29T10:48:13.651 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.668+0000 7f7df2f47640 10 monclient(hunting): picked mon.noname-c con 0x7f7dec0b17d0 addr v1:172.21.15.179:6789/0 2024-03-29T10:48:13.652 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.668+0000 7f7df2f47640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dec142c80 con 0x7f7dec1422b0 2024-03-29T10:48:13.652 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.668+0000 7f7df2f47640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dec0aae40 con 0x7f7dec0b17d0 2024-03-29T10:48:13.652 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.668+0000 7f7df2f47640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dec13fbd0 con 0x7f7dec142690 2024-03-29T10:48:13.652 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.668+0000 7f7df2f47640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:13.652 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.668+0000 7f7df2f47640 10 monclient(hunting): authenticate will time out at 1445.845732s 2024-03-29T10:48:13.654 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.671+0000 7f7df14e4640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f7dec0b17d0 0x7f7dec0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:56301/0 (socket says 172.21.15.57:56301) 2024-03-29T10:48:13.654 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.672+0000 7f7df14e4640 1 -- 172.21.15.57:0/3709409467 learned_addr learned my addr 172.21.15.57:0/3709409467 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:48:13.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.676+0000 7f7deb7fe640 1 -- 172.21.15.57:0/3709409467 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3238918009 0 0) 0x7f7dec13fbd0 con 0x7f7dec142690 2024-03-29T10:48:13.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.676+0000 7f7deb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:13.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.676+0000 7f7deb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:13.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.676+0000 7f7deb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:13.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.676+0000 7f7deb7fe640 10 monclient(hunting): my global_id is 4565 2024-03-29T10:48:13.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.677+0000 7f7deb7fe640 1 -- 172.21.15.57:0/3709409467 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dd4003520 con 0x7f7dec142690 2024-03-29T10:48:13.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.677+0000 7f7deb7fe640 1 -- 172.21.15.57:0/3709409467 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2296257775 0 0) 0x7f7dec142c80 con 0x7f7dec1422b0 2024-03-29T10:48:13.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.677+0000 7f7deb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:13.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.677+0000 7f7deb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:13.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.677+0000 7f7deb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:13.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.677+0000 7f7deb7fe640 10 monclient(hunting): my global_id is 4518 2024-03-29T10:48:13.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.677+0000 7f7deb7fe640 1 -- 172.21.15.57:0/3709409467 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dec13fbd0 con 0x7f7dec1422b0 2024-03-29T10:48:13.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.677+0000 7f7deb7fe640 1 -- 172.21.15.57:0/3709409467 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691598288 0 0) 0x7f7dec0aae40 con 0x7f7dec0b17d0 2024-03-29T10:48:13.661 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.677+0000 7f7deb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:13.662 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.677+0000 7f7deb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:13.662 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.677+0000 7f7deb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:13.662 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.677+0000 7f7deb7fe640 10 monclient(hunting): my global_id is 4516 2024-03-29T10:48:13.662 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.677+0000 7f7deb7fe640 1 -- 172.21.15.57:0/3709409467 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dec142c80 con 0x7f7dec0b17d0 2024-03-29T10:48:13.662 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.678+0000 7f7deb7fe640 1 -- 172.21.15.57:0/3709409467 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2849769492 0 0) 0x7f7dec142c80 con 0x7f7dec0b17d0 2024-03-29T10:48:13.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.678+0000 7f7deb7fe640 1 -- 172.21.15.57:0/3709409467 >> v1:172.21.15.57:6790/0 conn(0x7f7dec142690 legacy=0x7f7dec0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:13.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.678+0000 7f7deb7fe640 1 -- 172.21.15.57:0/3709409467 >> v1:172.21.15.57:6789/0 conn(0x7f7dec1422b0 legacy=0x7f7dec13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:13.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.678+0000 7f7deb7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:48:13.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.678+0000 7f7deb7fe640 1 monclient: found mon.noname-c 2024-03-29T10:48:13.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.678+0000 7f7deb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:13.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.678+0000 7f7deb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:48:13.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.678+0000 7f7deb7fe640 1 -- 172.21.15.57:0/3709409467 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dec151940 con 0x7f7dec0b17d0 2024-03-29T10:48:13.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.678+0000 7f7deb7fe640 10 monclient: _finish_auth 0 2024-03-29T10:48:13.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.678+0000 7f7deb7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:48:13.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.678+0000 7f7deb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:13.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.678+0000 7f7deb7fe640 1 -- 172.21.15.57:0/3709409467 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f7de4001530 con 0x7f7dec0b17d0 2024-03-29T10:48:13.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.678+0000 7f7deb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:13.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.678+0000 7f7deb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:13.663 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.678+0000 7f7deb7fe640 10 monclient: dump: 2024-03-29T10:48:13.663 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:13.664 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:13.664 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:13.664 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:13.664 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:13.664 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:13.664 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:13.664 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:13.664 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:13.664 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:13.664 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.678+0000 7f7deb7fe640 1 -- 172.21.15.57:0/3709409467 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7de4001690 con 0x7f7dec0b17d0 2024-03-29T10:48:13.664 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.678+0000 7f7deb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:13.664 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.678+0000 7f7deb7fe640 1 -- 172.21.15.57:0/3709409467 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f7de4001a60 con 0x7f7dec0b17d0 2024-03-29T10:48:13.664 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.678+0000 7f7deb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:13.664 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.678+0000 7f7deb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:48:13.664 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.678+0000 7f7deb7fe640 10 monclient: dump: 2024-03-29T10:48:13.664 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:13.665 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:13.665 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:13.665 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:13.665 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:13.665 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:13.665 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:13.665 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:13.665 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:13.665 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:13.665 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.679+0000 7f7df2f47640 5 monclient: authenticate success, global_id 4516 2024-03-29T10:48:13.665 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.679+0000 7f7df2f47640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:13.665 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.679+0000 7f7df2f47640 4 set_mon_vals no callback set 2024-03-29T10:48:13.665 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.680+0000 7f7df2f47640 10 monclient: shutdown 2024-03-29T10:48:13.665 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.680+0000 7f7df2f47640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:13.666 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.680+0000 7f7df2f47640 1 -- 172.21.15.57:0/3709409467 >> v1:172.21.15.179:6789/0 conn(0x7f7dec0b17d0 legacy=0x7f7dec0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:13.666 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.680+0000 7f7df2f47640 1 -- 172.21.15.57:0/3709409467 shutdown_connections 2024-03-29T10:48:13.666 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.680+0000 7f7df2f47640 1 -- 172.21.15.57:0/3709409467 >> 172.21.15.57:0/3709409467 conn(0x7f7dec066a00 msgr2=0x7f7dec066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:13.667 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.680+0000 7f7df2f47640 1 -- 172.21.15.57:0/3709409467 shutdown_connections 2024-03-29T10:48:13.667 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.680+0000 7f7df2f47640 1 -- 172.21.15.57:0/3709409467 wait complete. 2024-03-29T10:48:13.667 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.681+0000 7f7df2f47640 10 monclient: build_initial_monmap 2024-03-29T10:48:13.667 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.681+0000 7f7df2f47640 10 monclient: monmap: 2024-03-29T10:48:13.667 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:48:13.667 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:13.667 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:48:13.667 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:48:13.667 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:13.667 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:13.667 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:48:13.667 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:48:13.667 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:48:13.667 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:13.668 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.681+0000 7f7df2f47640 1 librados: starting msgr at 2024-03-29T10:48:13.668 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.681+0000 7f7df2f47640 1 librados: starting objecter 2024-03-29T10:48:13.668 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.684+0000 7f7df2f47640 1 Processor -- start 2024-03-29T10:48:13.670 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.687+0000 7f7df2f47640 1 -- start start 2024-03-29T10:48:13.670 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.687+0000 7f7df2f47640 1 librados: setting wanted keys 2024-03-29T10:48:13.670 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.687+0000 7f7df2f47640 1 librados: calling monclient init 2024-03-29T10:48:13.671 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.687+0000 7f7df2f47640 10 monclient: init 2024-03-29T10:48:13.673 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.688+0000 7f7df2f47640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:13.673 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.688+0000 7f7df2f47640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:48:13.673 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.688+0000 7f7df2f47640 10 monclient(hunting): picked mon.noname-c con 0x7f7dec0b17d0 addr v1:172.21.15.57:6790/0 2024-03-29T10:48:13.673 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.688+0000 7f7df2f47640 10 monclient(hunting): picked mon.noname-a con 0x7f7dec1422b0 addr v1:172.21.15.57:6789/0 2024-03-29T10:48:13.674 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.688+0000 7f7df2f47640 10 monclient(hunting): picked mon.noname-b con 0x7f7dec142690 addr v1:172.21.15.179:6789/0 2024-03-29T10:48:13.674 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.688+0000 7f7df2f47640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dec1323b0 con 0x7f7dec1422b0 2024-03-29T10:48:13.674 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.688+0000 7f7df2f47640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dec130390 con 0x7f7dec142690 2024-03-29T10:48:13.674 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.688+0000 7f7df2f47640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dec1327d0 con 0x7f7dec0b17d0 2024-03-29T10:48:13.674 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.688+0000 7f7df2f47640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:13.674 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.688+0000 7f7df2f47640 10 monclient(hunting): authenticate will time out at 1445.865719s 2024-03-29T10:48:13.676 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.691+0000 7f7df0ce3640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f7dec0b17d0 0x7f7dec12c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:58552/0 (socket says 172.21.15.57:58552) 2024-03-29T10:48:13.677 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.691+0000 7f7df0ce3640 1 -- 172.21.15.57:0/282955973 learned_addr learned my addr 172.21.15.57:0/282955973 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:48:13.677 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.691+0000 7f7de97fa640 10 client.?.objecter ms_handle_connect 0x7f7dec0b17d0 2024-03-29T10:48:13.677 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.692+0000 7f7de97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:13.677 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.692+0000 7f7de97fa640 10 client.?.objecter ms_handle_connect 0x7f7dec1422b0 2024-03-29T10:48:13.677 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.692+0000 7f7de97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:13.677 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.692+0000 7f7de97fa640 1 -- 172.21.15.57:0/282955973 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710765967 0 0) 0x7f7dec1327d0 con 0x7f7dec0b17d0 2024-03-29T10:48:13.677 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.692+0000 7f7de97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:48:13.677 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.692+0000 7f7de97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:13.677 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.692+0000 7f7de97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:13.677 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.692+0000 7f7de97fa640 10 monclient(hunting): my global_id is 4568 2024-03-29T10:48:13.677 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.692+0000 7f7de97fa640 1 -- 172.21.15.57:0/282955973 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dd8004eb0 con 0x7f7dec0b17d0 2024-03-29T10:48:13.677 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.693+0000 7f7de97fa640 1 -- 172.21.15.57:0/282955973 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863154392 0 0) 0x7f7dec1323b0 con 0x7f7dec1422b0 2024-03-29T10:48:13.677 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.693+0000 7f7de97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:48:13.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.693+0000 7f7de97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:13.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.693+0000 7f7de97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:13.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.693+0000 7f7de97fa640 10 monclient(hunting): my global_id is 4521 2024-03-29T10:48:13.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.693+0000 7f7de97fa640 1 -- 172.21.15.57:0/282955973 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dec1327d0 con 0x7f7dec1422b0 2024-03-29T10:48:13.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.693+0000 7f7de97fa640 1 -- 172.21.15.57:0/282955973 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1652931271 0 0) 0x7f7dd8004eb0 con 0x7f7dec0b17d0 2024-03-29T10:48:13.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.693+0000 7f7de97fa640 1 -- 172.21.15.57:0/282955973 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7dec1323b0 con 0x7f7dec0b17d0 2024-03-29T10:48:13.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.693+0000 7f7de97fa640 1 -- 172.21.15.57:0/282955973 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f7ddc001e00 con 0x7f7dec0b17d0 2024-03-29T10:48:13.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.693+0000 7f7de97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:13.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.693+0000 7f7de97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:13.678 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.693+0000 7f7de97fa640 10 monclient(hunting): dump: 2024-03-29T10:48:13.678 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:13.678 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:13.679 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:13.679 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:13.679 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:13.679 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:13.679 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:13.679 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:13.679 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:13.679 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:13.679 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.693+0000 7f7de97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:13.679 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.696+0000 7f7de97fa640 1 -- 172.21.15.57:0/282955973 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1500392912 0 0) 0x7f7dec1323b0 con 0x7f7dec0b17d0 2024-03-29T10:48:13.681 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.697+0000 7f7de97fa640 1 -- 172.21.15.57:0/282955973 >> v1:172.21.15.179:6789/0 conn(0x7f7dec142690 legacy=0x7f7dec1eea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:13.681 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.697+0000 7f7de97fa640 1 -- 172.21.15.57:0/282955973 >> v1:172.21.15.57:6789/0 conn(0x7f7dec1422b0 legacy=0x7f7dec12c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:13.681 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.697+0000 7f7de97fa640 10 monclient: _finish_hunting 0 2024-03-29T10:48:13.681 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.697+0000 7f7de97fa640 1 monclient: found mon.c 2024-03-29T10:48:13.681 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.697+0000 7f7de97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:13.681 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.697+0000 7f7de97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:13.681 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.698+0000 7f7de97fa640 1 -- 172.21.15.57:0/282955973 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dec132960 con 0x7f7dec0b17d0 2024-03-29T10:48:13.681 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.698+0000 7f7de97fa640 10 monclient: _finish_auth 0 2024-03-29T10:48:13.682 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.698+0000 7f7de97fa640 10 monclient: _check_auth_tickets 2024-03-29T10:48:13.682 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.698+0000 7f7de97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:13.682 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.698+0000 7f7df2f47640 5 monclient: authenticate success, global_id 4568 2024-03-29T10:48:13.682 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.698+0000 7f7df2f47640 10 monclient: _renew_subs 2024-03-29T10:48:13.682 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.698+0000 7f7df2f47640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:13.682 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.698+0000 7f7df2f47640 1 -- 172.21.15.57:0/282955973 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dec130a40 con 0x7f7dec0b17d0 2024-03-29T10:48:13.682 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.698+0000 7f7df2f47640 10 client.4568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:13.682 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.698+0000 7f7df2f47640 10 monclient: _renew_subs 2024-03-29T10:48:13.682 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.698+0000 7f7df2f47640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:13.682 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.698+0000 7f7df2f47640 1 -- 172.21.15.57:0/282955973 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dec12ff30 con 0x7f7dec0b17d0 2024-03-29T10:48:13.682 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.698+0000 7f7df2f47640 1 librados: init done 2024-03-29T10:48:13.682 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.698+0000 7f7de97fa640 10 client.4568.objecter ms_handle_connect 0x7f7dec142690 2024-03-29T10:48:13.682 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.698+0000 7f7de97fa640 10 client.4568.objecter resend_mon_ops 2024-03-29T10:48:13.683 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.698+0000 7f7de97fa640 1 -- 172.21.15.57:0/282955973 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ddc002020 con 0x7f7dec0b17d0 2024-03-29T10:48:13.683 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.698+0000 7f7de97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:13.683 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.698+0000 7f7de97fa640 1 -- 172.21.15.57:0/282955973 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f7ddc003de0 con 0x7f7dec0b17d0 2024-03-29T10:48:13.683 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.698+0000 7f7de97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:13.683 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.698+0000 7f7de97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:13.683 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.698+0000 7f7de97fa640 10 monclient: dump: 2024-03-29T10:48:13.683 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:13.683 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:13.683 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:13.683 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:13.683 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:13.683 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:13.683 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:13.683 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:13.684 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:13.684 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:13.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.698+0000 7f7dea7fc640 4 set_mon_vals no callback set 2024-03-29T10:48:13.685 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.701+0000 7f7df2f47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:13.685 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.701+0000 7f7df2f47640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:13.686 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.701+0000 7f7df2f47640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:13.686 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.701+0000 7f7df2f47640 1 -- 172.21.15.57:0/282955973 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dcc005080 con 0x7f7dec0b17d0 2024-03-29T10:48:13.686 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.703+0000 7f7de97fa640 1 -- 172.21.15.57:0/282955973 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f7ddc003f40 con 0x7f7dec0b17d0 2024-03-29T10:48:13.686 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.703+0000 7f7de97fa640 1 -- 172.21.15.57:0/282955973 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f7ddc084ea0 con 0x7f7dec0b17d0 2024-03-29T10:48:13.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.703+0000 7f7de97fa640 10 client.4568.objecter ms_dispatch 0x7f7dec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:13.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.703+0000 7f7de97fa640 3 client.4568.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:13.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.703+0000 7f7de97fa640 3 client.4568.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:13.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.703+0000 7f7debfff640 10 monclient: get_auth_request con 0x7f7dd806db60 auth_method 0 2024-03-29T10:48:13.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.704+0000 7f7de97fa640 20 client.4568.objecter dump_active .. 0 homeless 2024-03-29T10:48:13.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.704+0000 7f7de97fa640 10 client.4568.objecter ms_handle_connect 0x7f7dd806db60 2024-03-29T10:48:13.695 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.712+0000 7f7de97fa640 1 -- 172.21.15.57:0/282955973 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7ddc0505b0 con 0x7f7dec0b17d0 2024-03-29T10:48:13.695 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.712+0000 7f7de97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:13.695 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:13.712+0000 7f7de97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:14.042+0000 7f7df2f47640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:14.042+0000 7f7df2f47640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:14.042+0000 7f7df2f47640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:14.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:14.042+0000 7f7df2f47640 1 -- 172.21.15.57:0/282955973 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7dcc005330 con 0x7f7dec0b17d0 2024-03-29T10:48:14.027 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:14.044+0000 7f7de97fa640 1 -- 172.21.15.57:0/282955973 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f7ddc0526b0 con 0x7f7dec0b17d0 2024-03-29T10:48:14.027 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:14.044+0000 7f7de97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:14.027 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:14.044+0000 7f7de97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:14.027 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:48:14.027 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:14.047+0000 7f7df2f47640 10 librados: watch_flush enter 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:14.047+0000 7f7df2f47640 10 librados: watch_flush exit 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:14.047+0000 7f7df2f47640 20 client.4568.objecter shutdown clearing up homeless session... 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:14.047+0000 7f7df2f47640 10 client.4568.objecter successfully canceled tick 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:14.047+0000 7f7df2f47640 1 -- 172.21.15.57:0/282955973 >> v1:172.21.15.179:6816/2698295936 conn(0x7f7dd806db60 legacy=0x7f7dd8070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:14.047+0000 7f7df2f47640 10 monclient: shutdown 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:14.047+0000 7f7df2f47640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:14.047+0000 7f7df2f47640 1 -- 172.21.15.57:0/282955973 >> v1:172.21.15.57:6790/0 conn(0x7f7dec0b17d0 legacy=0x7f7dec12c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:14.047+0000 7f7df2f47640 1 -- 172.21.15.57:0/282955973 shutdown_connections 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:14.047+0000 7f7df2f47640 1 -- 172.21.15.57:0/282955973 >> 172.21.15.57:0/282955973 conn(0x7f7dec066a00 msgr2=0x7f7dec13df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:14.048+0000 7f7df2f47640 1 -- 172.21.15.57:0/282955973 shutdown_connections 2024-03-29T10:48:14.031 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:14.048+0000 7f7df2f47640 1 -- 172.21.15.57:0/282955973 wait complete. 2024-03-29T10:48:14.032 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:14.048+0000 7f7df2f47640 1 librados: shutdown 2024-03-29T10:48:14.047 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:20.854 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:21.005 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.023+0000 7f021c8ae640 10 monclient: get_monmap_and_config 2024-03-29T10:48:21.005 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.023+0000 7f021c8ae640 10 monclient: build_initial_monmap 2024-03-29T10:48:21.006 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.023+0000 7f021c8ae640 10 monclient: monmap: 2024-03-29T10:48:21.006 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:48:21.006 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:21.006 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:48:21.024335+0000 2024-03-29T10:48:21.006 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:48:21.024335+0000 2024-03-29T10:48:21.006 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:21.006 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:21.006 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:48:21.006 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:48:21.006 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:48:21.006 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:21.007 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.024+0000 7f021c8ae640 1 Processor -- start 2024-03-29T10:48:21.008 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.024+0000 7f021c8ae640 1 -- start start 2024-03-29T10:48:21.008 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.024+0000 7f021c8ae640 10 monclient: init 2024-03-29T10:48:21.008 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.025+0000 7f021c8ae640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:21.008 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.025+0000 7f021c8ae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:48:21.008 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.025+0000 7f021c8ae640 10 monclient(hunting): picked mon.noname-c con 0x7f0210038c60 addr v1:172.21.15.179:6789/0 2024-03-29T10:48:21.008 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.026+0000 7f021c8ae640 10 monclient(hunting): picked mon.noname-a con 0x7f02100324b0 addr v1:172.21.15.57:6789/0 2024-03-29T10:48:21.008 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.026+0000 7f021c8ae640 10 monclient(hunting): picked mon.noname-b con 0x7f02100d11e0 addr v1:172.21.15.57:6790/0 2024-03-29T10:48:21.008 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.026+0000 7f021659c640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f02100324b0 0x7f02100cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:33258/0 (socket says 172.21.15.57:33258) 2024-03-29T10:48:21.009 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.026+0000 7f021659c640 1 -- 172.21.15.57:0/3792039400 learned_addr learned my addr 172.21.15.57:0/3792039400 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:48:21.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.026+0000 7f0216d9d640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f0210038c60 0x7f02100c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:52190/0 (socket says 172.21.15.57:52190) 2024-03-29T10:48:21.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.026+0000 7f021c8ae640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02100caf80 con 0x7f02100324b0 2024-03-29T10:48:21.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.026+0000 7f021c8ae640 1 -- 172.21.15.57:0/3792039400 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0210039850 con 0x7f0210038c60 2024-03-29T10:48:21.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.026+0000 7f021c8ae640 1 -- 172.21.15.57:0/3792039400 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02100c78d0 con 0x7f02100d11e0 2024-03-29T10:48:21.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.026+0000 7f021c8ae640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:21.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.027+0000 7f021c8ae640 10 monclient(hunting): authenticate will time out at 1453.203081s 2024-03-29T10:48:21.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.027+0000 7f0215d9b640 1 -- 172.21.15.57:0/3792039400 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 70197713 0 0) 0x7f0210039850 con 0x7f0210038c60 2024-03-29T10:48:21.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.027+0000 7f0215d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:48:21.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.027+0000 7f0215d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:21.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.027+0000 7f0215d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:21.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.027+0000 7f0215d9b640 10 monclient(hunting): my global_id is 4522 2024-03-29T10:48:21.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.027+0000 7f0215d9b640 1 -- 172.21.15.57:0/3792039400 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0200003520 con 0x7f0210038c60 2024-03-29T10:48:21.010 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.027+0000 7f0215d9b640 1 -- 172.21.15.57:0/3792039400 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506997481 0 0) 0x7f02100c78d0 con 0x7f02100d11e0 2024-03-29T10:48:21.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.027+0000 7f0215d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:48:21.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.027+0000 7f0215d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:21.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.027+0000 7f0215d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:21.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.027+0000 7f0215d9b640 10 monclient(hunting): my global_id is 4571 2024-03-29T10:48:21.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.027+0000 7f0215d9b640 1 -- 172.21.15.57:0/3792039400 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0210039850 con 0x7f02100d11e0 2024-03-29T10:48:21.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.027+0000 7f0215d9b640 1 -- 172.21.15.57:0/3792039400 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971065605 0 0) 0x7f02100caf80 con 0x7f02100324b0 2024-03-29T10:48:21.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.027+0000 7f0215d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:48:21.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.027+0000 7f0215d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:21.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.027+0000 7f0215d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:21.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.027+0000 7f0215d9b640 10 monclient(hunting): my global_id is 4524 2024-03-29T10:48:21.011 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.027+0000 7f0215d9b640 1 -- 172.21.15.57:0/3792039400 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02100c78d0 con 0x7f02100324b0 2024-03-29T10:48:21.013 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f0215d9b640 1 -- 172.21.15.57:0/3792039400 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 335677511 0 0) 0x7f0210039850 con 0x7f02100d11e0 2024-03-29T10:48:21.013 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f0215d9b640 1 -- 172.21.15.57:0/3792039400 >> v1:172.21.15.179:6789/0 conn(0x7f0210038c60 legacy=0x7f02100c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:21.013 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f0215d9b640 1 -- 172.21.15.57:0/3792039400 >> v1:172.21.15.57:6789/0 conn(0x7f02100324b0 legacy=0x7f02100cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:21.013 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f0215d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:48:21.013 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f0215d9b640 1 monclient: found mon.noname-b 2024-03-29T10:48:21.013 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f0215d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:21.013 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f0215d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:48:21.014 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f0215d9b640 1 -- 172.21.15.57:0/3792039400 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02100c7a60 con 0x7f02100d11e0 2024-03-29T10:48:21.014 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f0215d9b640 10 monclient: _finish_auth 0 2024-03-29T10:48:21.014 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f0215d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:48:21.014 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f0215d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:21.014 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f021c8ae640 5 monclient: authenticate success, global_id 4571 2024-03-29T10:48:21.014 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f021c8ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:21.014 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f0215d9b640 1 -- 172.21.15.57:0/3792039400 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f020c001690 con 0x7f02100d11e0 2024-03-29T10:48:21.014 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f0215d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:21.014 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f0215d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:21.014 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f0215d9b640 10 monclient: dump: 2024-03-29T10:48:21.014 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:21.014 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:21.014 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:21.014 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:21.015 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:21.015 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:21.015 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:21.015 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:21.015 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:21.015 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:21.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f021c8ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:21.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f0215d9b640 1 -- 172.21.15.57:0/3792039400 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f020c0018e0 con 0x7f02100d11e0 2024-03-29T10:48:21.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f0215d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:21.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f0215d9b640 1 -- 172.21.15.57:0/3792039400 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f020c001c70 con 0x7f02100d11e0 2024-03-29T10:48:21.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f0215d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:21.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f0215d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:21.015 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f0215d9b640 10 monclient: dump: 2024-03-29T10:48:21.015 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:21.016 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:21.016 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:21.016 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:21.016 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:21.016 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:21.016 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:21.016 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:21.016 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:21.016 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:21.016 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f021c8ae640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:21.016 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.028+0000 7f021c8ae640 4 set_mon_vals no callback set 2024-03-29T10:48:21.016 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.029+0000 7f021c8ae640 10 monclient: shutdown 2024-03-29T10:48:21.016 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.029+0000 7f021c8ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:21.016 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.029+0000 7f021c8ae640 1 -- 172.21.15.57:0/3792039400 >> v1:172.21.15.57:6790/0 conn(0x7f02100d11e0 legacy=0x7f02100d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:21.017 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.030+0000 7f021c8ae640 1 -- 172.21.15.57:0/3792039400 shutdown_connections 2024-03-29T10:48:21.017 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.030+0000 7f021c8ae640 1 -- 172.21.15.57:0/3792039400 >> 172.21.15.57:0/3792039400 conn(0x7f0210005c20 msgr2=0x7f0210006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:21.017 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.030+0000 7f021c8ae640 1 -- 172.21.15.57:0/3792039400 shutdown_connections 2024-03-29T10:48:21.017 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.030+0000 7f021c8ae640 1 -- 172.21.15.57:0/3792039400 wait complete. 2024-03-29T10:48:21.017 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.031+0000 7f021c8ae640 10 monclient: build_initial_monmap 2024-03-29T10:48:21.017 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.031+0000 7f021c8ae640 10 monclient: monmap: 2024-03-29T10:48:21.017 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:48:21.017 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:21.017 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:48:21.017 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:48:21.017 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:21.017 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:21.017 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:48:21.017 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:48:21.018 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:48:21.018 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:21.021 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.032+0000 7f021c8ae640 1 librados: starting msgr at 2024-03-29T10:48:21.021 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.032+0000 7f021c8ae640 1 librados: starting objecter 2024-03-29T10:48:21.021 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.032+0000 7f021c8ae640 1 Processor -- start 2024-03-29T10:48:21.021 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.032+0000 7f021c8ae640 1 -- start start 2024-03-29T10:48:21.021 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.032+0000 7f021c8ae640 1 librados: setting wanted keys 2024-03-29T10:48:21.021 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.032+0000 7f021c8ae640 1 librados: calling monclient init 2024-03-29T10:48:21.021 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.032+0000 7f021c8ae640 10 monclient: init 2024-03-29T10:48:21.021 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.033+0000 7f021c8ae640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:21.021 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.033+0000 7f021c8ae640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:48:21.022 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.033+0000 7f021c8ae640 10 monclient(hunting): picked mon.noname-b con 0x7f02100324b0 addr v1:172.21.15.179:6789/0 2024-03-29T10:48:21.022 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.033+0000 7f021c8ae640 10 monclient(hunting): picked mon.noname-c con 0x7f0210038c60 addr v1:172.21.15.57:6790/0 2024-03-29T10:48:21.022 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.033+0000 7f021c8ae640 10 monclient(hunting): picked mon.noname-a con 0x7f02100d11e0 addr v1:172.21.15.57:6789/0 2024-03-29T10:48:21.022 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.033+0000 7f021c8ae640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0210034870 con 0x7f02100d11e0 2024-03-29T10:48:21.022 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.034+0000 7f021c8ae640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02100c1670 con 0x7f02100324b0 2024-03-29T10:48:21.022 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.034+0000 7f021c8ae640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0210037200 con 0x7f0210038c60 2024-03-29T10:48:21.022 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.034+0000 7f021c8ae640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:21.022 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.034+0000 7f021c8ae640 10 monclient(hunting): authenticate will time out at 1453.210050s 2024-03-29T10:48:21.022 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.034+0000 7f021759e640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f02100d11e0 0x7f0210167710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:33732/0 (socket says 172.21.15.57:33732) 2024-03-29T10:48:21.022 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.034+0000 7f021759e640 1 -- 172.21.15.57:0/2361150331 learned_addr learned my addr 172.21.15.57:0/2361150331 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:48:21.022 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.034+0000 7f01ff7fe640 10 client.?.objecter ms_handle_connect 0x7f02100d11e0 2024-03-29T10:48:21.022 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.034+0000 7f01ff7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:21.023 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.034+0000 7f01ff7fe640 10 client.?.objecter ms_handle_connect 0x7f0210038c60 2024-03-29T10:48:21.023 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.034+0000 7f01ff7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:21.023 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.034+0000 7f01ff7fe640 10 client.?.objecter ms_handle_connect 0x7f02100324b0 2024-03-29T10:48:21.023 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.034+0000 7f01ff7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:21.023 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.034+0000 7f01ff7fe640 1 -- 172.21.15.57:0/2361150331 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510911751 0 0) 0x7f0210034870 con 0x7f02100d11e0 2024-03-29T10:48:21.023 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.034+0000 7f01ff7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:21.023 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.034+0000 7f01ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:21.023 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.034+0000 7f01ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:21.023 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.034+0000 7f01ff7fe640 10 monclient(hunting): my global_id is 4527 2024-03-29T10:48:21.023 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.034+0000 7f01ff7fe640 1 -- 172.21.15.57:0/2361150331 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0204004eb0 con 0x7f02100d11e0 2024-03-29T10:48:21.023 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.035+0000 7f01ff7fe640 1 -- 172.21.15.57:0/2361150331 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 216264517 0 0) 0x7f0204004eb0 con 0x7f02100d11e0 2024-03-29T10:48:21.023 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.035+0000 7f01ff7fe640 1 -- 172.21.15.57:0/2361150331 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0204005300 con 0x7f02100d11e0 2024-03-29T10:48:21.023 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.035+0000 7f01ff7fe640 1 -- 172.21.15.57:0/2361150331 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f020c001bf0 con 0x7f02100d11e0 2024-03-29T10:48:21.023 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.035+0000 7f01ff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:21.024 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.035+0000 7f01ff7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:21.024 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.035+0000 7f01ff7fe640 10 monclient(hunting): dump: 2024-03-29T10:48:21.024 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:21.024 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:21.024 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:21.024 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:21.024 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:21.024 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:21.024 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:21.024 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:21.024 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:21.024 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:21.024 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.035+0000 7f01ff7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:21.024 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.035+0000 7f01ff7fe640 1 -- 172.21.15.57:0/2361150331 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 643383378 0 0) 0x7f0210037200 con 0x7f0210038c60 2024-03-29T10:48:21.025 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.035+0000 7f01ff7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:21.025 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.035+0000 7f01ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:21.025 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.035+0000 7f01ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:21.025 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.035+0000 7f01ff7fe640 10 monclient(hunting): my global_id is 4574 2024-03-29T10:48:21.025 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.035+0000 7f01ff7fe640 1 -- 172.21.15.57:0/2361150331 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0204005880 con 0x7f0210038c60 2024-03-29T10:48:21.025 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f01ff7fe640 1 -- 172.21.15.57:0/2361150331 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1359955618 0 0) 0x7f0204005300 con 0x7f02100d11e0 2024-03-29T10:48:21.025 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f01ff7fe640 1 -- 172.21.15.57:0/2361150331 >> v1:172.21.15.57:6790/0 conn(0x7f0210038c60 legacy=0x7f0210035410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:21.025 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f01ff7fe640 1 -- 172.21.15.57:0/2361150331 >> v1:172.21.15.179:6789/0 conn(0x7f02100324b0 legacy=0x7f0210034d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:21.025 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f01ff7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:48:21.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f01ff7fe640 1 monclient: found mon.a 2024-03-29T10:48:21.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f01ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:21.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f01ff7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:48:21.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f01ff7fe640 1 -- 172.21.15.57:0/2361150331 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0210030290 con 0x7f02100d11e0 2024-03-29T10:48:21.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f01ff7fe640 10 monclient: _finish_auth 0 2024-03-29T10:48:21.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f01ff7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:48:21.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f01ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:21.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f021c8ae640 5 monclient: authenticate success, global_id 4527 2024-03-29T10:48:21.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f021c8ae640 10 monclient: _renew_subs 2024-03-29T10:48:21.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f021c8ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:48:21.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f021c8ae640 1 -- 172.21.15.57:0/2361150331 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02100c0310 con 0x7f02100d11e0 2024-03-29T10:48:21.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f021c8ae640 10 client.4527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:21.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f021c8ae640 10 monclient: _renew_subs 2024-03-29T10:48:21.026 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f021c8ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:48:21.027 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f021c8ae640 1 -- 172.21.15.57:0/2361150331 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02100c1fe0 con 0x7f02100d11e0 2024-03-29T10:48:21.027 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f021c8ae640 1 librados: init done 2024-03-29T10:48:21.027 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f01ff7fe640 1 -- 172.21.15.57:0/2361150331 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f020c0038a0 con 0x7f02100d11e0 2024-03-29T10:48:21.027 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f01ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:21.027 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f01ff7fe640 1 -- 172.21.15.57:0/2361150331 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f020c001bf0 con 0x7f02100d11e0 2024-03-29T10:48:21.027 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f01ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:21.027 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f01ff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:21.027 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f01ff7fe640 10 monclient: dump: 2024-03-29T10:48:21.027 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:21.027 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:21.027 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:21.027 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:21.027 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:21.027 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:21.028 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:21.028 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:21.028 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:21.028 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:21.028 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.036+0000 7f021559a640 4 set_mon_vals no callback set 2024-03-29T10:48:21.028 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.038+0000 7f021c8ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:21.028 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.038+0000 7f021c8ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:21.028 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.038+0000 7f021c8ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:48:21.028 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.038+0000 7f021c8ae640 1 -- 172.21.15.57:0/2361150331 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0210007b70 con 0x7f02100d11e0 2024-03-29T10:48:21.029 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.040+0000 7f01ff7fe640 1 -- 172.21.15.57:0/2361150331 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f020c003b90 con 0x7f02100d11e0 2024-03-29T10:48:21.029 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.041+0000 7f0216d9d640 10 monclient: get_auth_request con 0x7f020406e4f0 auth_method 0 2024-03-29T10:48:21.029 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.041+0000 7f01ff7fe640 1 -- 172.21.15.57:0/2361150331 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f020c054390 con 0x7f02100d11e0 2024-03-29T10:48:21.029 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.041+0000 7f01ff7fe640 10 client.4527.objecter ms_dispatch 0x7f0218000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:21.029 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.041+0000 7f01ff7fe640 3 client.4527.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:21.029 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.041+0000 7f01ff7fe640 3 client.4527.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:21.029 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.041+0000 7f01ff7fe640 20 client.4527.objecter dump_active .. 0 homeless 2024-03-29T10:48:21.029 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.041+0000 7f01ff7fe640 10 client.4527.objecter ms_handle_connect 0x7f020406e4f0 2024-03-29T10:48:21.032 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.051+0000 7f01ff7fe640 1 -- 172.21.15.57:0/2361150331 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f020c050630 con 0x7f02100d11e0 2024-03-29T10:48:21.033 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.051+0000 7f01ff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:21.033 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.051+0000 7f01ff7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:21.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.449+0000 7f021c8ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:21.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.449+0000 7f021c8ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:21.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.449+0000 7f021c8ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:48:21.431 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.449+0000 7f021c8ae640 1 -- 172.21.15.57:0/2361150331 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02100c1470 con 0x7f02100d11e0 2024-03-29T10:48:21.432 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.451+0000 7f01ff7fe640 1 -- 172.21.15.57:0/2361150331 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f020c0507d0 con 0x7f02100d11e0 2024-03-29T10:48:21.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.451+0000 7f01ff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:21.433 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.451+0000 7f01ff7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:21.433 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:48:21.433 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:21.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.454+0000 7f021c8ae640 10 librados: watch_flush enter 2024-03-29T10:48:21.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.454+0000 7f021c8ae640 10 librados: watch_flush exit 2024-03-29T10:48:21.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.454+0000 7f021c8ae640 20 client.4527.objecter shutdown clearing up homeless session... 2024-03-29T10:48:21.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.454+0000 7f021c8ae640 10 client.4527.objecter successfully canceled tick 2024-03-29T10:48:21.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.454+0000 7f021c8ae640 1 -- 172.21.15.57:0/2361150331 >> v1:172.21.15.179:6816/2698295936 conn(0x7f020406e4f0 legacy=0x7f02040709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:21.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.454+0000 7f021c8ae640 10 monclient: shutdown 2024-03-29T10:48:21.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.454+0000 7f021c8ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:21.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.454+0000 7f021c8ae640 1 -- 172.21.15.57:0/2361150331 >> v1:172.21.15.57:6789/0 conn(0x7f02100d11e0 legacy=0x7f0210167710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:21.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.455+0000 7f021c8ae640 1 -- 172.21.15.57:0/2361150331 shutdown_connections 2024-03-29T10:48:21.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.455+0000 7f021c8ae640 1 -- 172.21.15.57:0/2361150331 >> 172.21.15.57:0/2361150331 conn(0x7f0210005c20 msgr2=0x7f02100043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:21.438 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.455+0000 7f021c8ae640 1 -- 172.21.15.57:0/2361150331 shutdown_connections 2024-03-29T10:48:21.438 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.455+0000 7f021c8ae640 1 -- 172.21.15.57:0/2361150331 wait complete. 2024-03-29T10:48:21.438 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:21.455+0000 7f021c8ae640 1 librados: shutdown 2024-03-29T10:48:21.452 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:28.260 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:28.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.409+0000 7f81ebac1640 10 monclient: get_monmap_and_config 2024-03-29T10:48:28.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.409+0000 7f81ebac1640 10 monclient: build_initial_monmap 2024-03-29T10:48:28.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.409+0000 7f81ebac1640 10 monclient: monmap: 2024-03-29T10:48:28.392 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:48:28.392 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:28.392 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:48:28.409994+0000 2024-03-29T10:48:28.392 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:48:28.409994+0000 2024-03-29T10:48:28.392 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:28.392 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:28.392 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:48:28.392 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:48:28.392 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:48:28.392 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:28.395 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.410+0000 7f81ebac1640 1 Processor -- start 2024-03-29T10:48:28.395 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.410+0000 7f81ebac1640 1 -- start start 2024-03-29T10:48:28.395 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.410+0000 7f81ebac1640 10 monclient: init 2024-03-29T10:48:28.395 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.411+0000 7f81ebac1640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:28.395 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.411+0000 7f81ebac1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:48:28.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.411+0000 7f81ebac1640 10 monclient(hunting): picked mon.noname-b con 0x7f81e41457f0 addr v1:172.21.15.57:6790/0 2024-03-29T10:48:28.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.411+0000 7f81ebac1640 10 monclient(hunting): picked mon.noname-a con 0x7f81e413f650 addr v1:172.21.15.57:6789/0 2024-03-29T10:48:28.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.411+0000 7f81ebac1640 10 monclient(hunting): picked mon.noname-c con 0x7f81e40a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T10:48:28.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.411+0000 7f81ebac1640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e40aaef0 con 0x7f81e413f650 2024-03-29T10:48:28.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.411+0000 7f81ebac1640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e4145bd0 con 0x7f81e40a9d70 2024-03-29T10:48:28.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.411+0000 7f81ebac1640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e4145eb0 con 0x7f81e41457f0 2024-03-29T10:48:28.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.411+0000 7f81ebac1640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:28.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.411+0000 7f81ebac1640 10 monclient(hunting): authenticate will time out at 1460.587548s 2024-03-29T10:48:28.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.411+0000 7f81e985d640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f81e41457f0 0x7f81e413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:39802/0 (socket says 172.21.15.57:39802) 2024-03-29T10:48:28.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.411+0000 7f81e985d640 1 -- 172.21.15.57:0/1941152381 learned_addr learned my addr 172.21.15.57:0/1941152381 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:48:28.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.412+0000 7f81e885b640 1 -- 172.21.15.57:0/1941152381 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 201854271 0 0) 0x7f81e4145eb0 con 0x7f81e41457f0 2024-03-29T10:48:28.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.412+0000 7f81e885b640 10 monclient(hunting): _negotiate 2024-03-29T10:48:28.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.412+0000 7f81e885b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:28.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.412+0000 7f81e885b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:28.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.412+0000 7f81e885b640 10 monclient(hunting): my global_id is 4577 2024-03-29T10:48:28.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.412+0000 7f81e885b640 1 -- 172.21.15.57:0/1941152381 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81d0003520 con 0x7f81e41457f0 2024-03-29T10:48:28.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.412+0000 7f81e885b640 1 -- 172.21.15.57:0/1941152381 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3591307354 0 0) 0x7f81d0003520 con 0x7f81e41457f0 2024-03-29T10:48:28.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.412+0000 7f81e885b640 1 -- 172.21.15.57:0/1941152381 >> v1:172.21.15.179:6789/0 conn(0x7f81e40a9d70 legacy=0x7f81e414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:28.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.412+0000 7f81e885b640 1 -- 172.21.15.57:0/1941152381 >> v1:172.21.15.57:6789/0 conn(0x7f81e413f650 legacy=0x7f81e40b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:28.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.413+0000 7f81e885b640 10 monclient: _finish_hunting 0 2024-03-29T10:48:28.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.413+0000 7f81e885b640 1 monclient: found mon.noname-b 2024-03-29T10:48:28.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.413+0000 7f81e885b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:28.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.413+0000 7f81e885b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:48:28.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.413+0000 7f81e885b640 1 -- 172.21.15.57:0/1941152381 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e4146340 con 0x7f81e41457f0 2024-03-29T10:48:28.398 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.413+0000 7f81e885b640 10 monclient: _finish_auth 0 2024-03-29T10:48:28.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.413+0000 7f81e885b640 10 monclient: _check_auth_tickets 2024-03-29T10:48:28.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.413+0000 7f81e885b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:28.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.413+0000 7f81ebac1640 5 monclient: authenticate success, global_id 4577 2024-03-29T10:48:28.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.413+0000 7f81ebac1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:28.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.413+0000 7f81e885b640 1 -- 172.21.15.57:0/1941152381 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f81e0001690 con 0x7f81e41457f0 2024-03-29T10:48:28.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.413+0000 7f81e885b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:28.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.413+0000 7f81e885b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:28.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.413+0000 7f81e885b640 10 monclient: dump: 2024-03-29T10:48:28.399 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:28.399 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:28.399 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:28.399 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:28.399 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:28.400 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:28.400 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:28.400 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:28.400 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:28.400 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:28.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.413+0000 7f81ebac1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:28.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.414+0000 7f81e885b640 1 -- 172.21.15.57:0/1941152381 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81e0001920 con 0x7f81e41457f0 2024-03-29T10:48:28.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.414+0000 7f81e885b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:28.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.414+0000 7f81ebac1640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:28.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.414+0000 7f81ebac1640 4 set_mon_vals no callback set 2024-03-29T10:48:28.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.414+0000 7f81ebac1640 10 monclient: shutdown 2024-03-29T10:48:28.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.414+0000 7f81ebac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:28.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.414+0000 7f81ebac1640 1 -- 172.21.15.57:0/1941152381 >> v1:172.21.15.57:6790/0 conn(0x7f81e41457f0 legacy=0x7f81e413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:28.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.415+0000 7f81ebac1640 1 -- 172.21.15.57:0/1941152381 shutdown_connections 2024-03-29T10:48:28.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.415+0000 7f81ebac1640 1 -- 172.21.15.57:0/1941152381 >> 172.21.15.57:0/1941152381 conn(0x7f81e4066a00 msgr2=0x7f81e4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:28.401 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.416+0000 7f81ebac1640 1 -- 172.21.15.57:0/1941152381 shutdown_connections 2024-03-29T10:48:28.401 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.416+0000 7f81ebac1640 1 -- 172.21.15.57:0/1941152381 wait complete. 2024-03-29T10:48:28.401 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.417+0000 7f81ebac1640 10 monclient: build_initial_monmap 2024-03-29T10:48:28.401 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.417+0000 7f81ebac1640 10 monclient: monmap: 2024-03-29T10:48:28.401 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:48:28.401 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:28.401 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:48:28.401 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:48:28.401 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:28.401 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:28.401 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:48:28.401 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:48:28.401 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:48:28.401 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:28.408 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.417+0000 7f81ebac1640 1 librados: starting msgr at 2024-03-29T10:48:28.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.417+0000 7f81ebac1640 1 librados: starting objecter 2024-03-29T10:48:28.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.417+0000 7f81ebac1640 1 Processor -- start 2024-03-29T10:48:28.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.417+0000 7f81ebac1640 1 -- start start 2024-03-29T10:48:28.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.417+0000 7f81ebac1640 1 librados: setting wanted keys 2024-03-29T10:48:28.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.417+0000 7f81ebac1640 1 librados: calling monclient init 2024-03-29T10:48:28.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.417+0000 7f81ebac1640 10 monclient: init 2024-03-29T10:48:28.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.418+0000 7f81ebac1640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:28.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.418+0000 7f81ebac1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:48:28.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.418+0000 7f81ebac1640 10 monclient(hunting): picked mon.noname-b con 0x7f81e40a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T10:48:28.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.418+0000 7f81ebac1640 10 monclient(hunting): picked mon.noname-a con 0x7f81e413f650 addr v1:172.21.15.57:6789/0 2024-03-29T10:48:28.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.418+0000 7f81ebac1640 10 monclient(hunting): picked mon.noname-c con 0x7f81e41457f0 addr v1:172.21.15.57:6790/0 2024-03-29T10:48:28.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.418+0000 7f81ebac1640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e41db9f0 con 0x7f81e413f650 2024-03-29T10:48:28.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.418+0000 7f81ebac1640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e4138700 con 0x7f81e40a9d70 2024-03-29T10:48:28.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.418+0000 7f81ebac1640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e41db6c0 con 0x7f81e41457f0 2024-03-29T10:48:28.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.418+0000 7f81ebac1640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:28.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.418+0000 7f81ebac1640 10 monclient(hunting): authenticate will time out at 1460.594765s 2024-03-29T10:48:28.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.418+0000 7f81ea05e640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f81e41457f0 0x7f81e4137ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:40508/0 (socket says 172.21.15.57:40508) 2024-03-29T10:48:28.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.418+0000 7f81ea05e640 1 -- 172.21.15.57:0/4216589193 learned_addr learned my addr 172.21.15.57:0/4216589193 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:48:28.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.419+0000 7f81da7fc640 10 client.?.objecter ms_handle_connect 0x7f81e41457f0 2024-03-29T10:48:28.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.419+0000 7f81da7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:28.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.419+0000 7f81da7fc640 1 -- 172.21.15.57:0/4216589193 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765214606 0 0) 0x7f81e41db6c0 con 0x7f81e41457f0 2024-03-29T10:48:28.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.419+0000 7f81da7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:28.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.419+0000 7f81da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:28.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.419+0000 7f81da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:28.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.419+0000 7f81da7fc640 10 monclient(hunting): my global_id is 4580 2024-03-29T10:48:28.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.419+0000 7f81da7fc640 1 -- 172.21.15.57:0/4216589193 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81c8004eb0 con 0x7f81e41457f0 2024-03-29T10:48:28.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.419+0000 7f81da7fc640 1 -- 172.21.15.57:0/4216589193 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3879936654 0 0) 0x7f81c8004eb0 con 0x7f81e41457f0 2024-03-29T10:48:28.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.419+0000 7f81da7fc640 1 -- 172.21.15.57:0/4216589193 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81e41db6c0 con 0x7f81e41457f0 2024-03-29T10:48:28.411 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.419+0000 7f81da7fc640 1 -- 172.21.15.57:0/4216589193 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f81dc001d30 con 0x7f81e41457f0 2024-03-29T10:48:28.411 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.419+0000 7f81da7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:28.411 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.419+0000 7f81da7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:28.411 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.419+0000 7f81da7fc640 10 monclient(hunting): dump: 2024-03-29T10:48:28.411 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:28.411 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:28.411 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:28.412 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:28.412 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:28.412 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:28.412 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:28.412 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:28.412 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:28.412 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:28.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.419+0000 7f81da7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:28.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.419+0000 7f81da7fc640 1 -- 172.21.15.57:0/4216589193 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2164857889 0 0) 0x7f81e41db6c0 con 0x7f81e41457f0 2024-03-29T10:48:28.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.420+0000 7f81da7fc640 1 -- 172.21.15.57:0/4216589193 >> v1:172.21.15.179:6789/0 conn(0x7f81e40a9d70 legacy=0x7f81e41dbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:28.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.420+0000 7f81da7fc640 1 -- 172.21.15.57:0/4216589193 >> v1:172.21.15.57:6789/0 conn(0x7f81e413f650 legacy=0x7f81e41378e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:48:28.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.421+0000 7f81da7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:48:28.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.421+0000 7f81da7fc640 1 monclient: found mon.c 2024-03-29T10:48:28.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.421+0000 7f81da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:28.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.421+0000 7f81da7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:28.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.421+0000 7f81da7fc640 1 -- 172.21.15.57:0/4216589193 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e41dc590 con 0x7f81e41457f0 2024-03-29T10:48:28.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.421+0000 7f81da7fc640 10 monclient: _finish_auth 0 2024-03-29T10:48:28.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.421+0000 7f81da7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:48:28.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.421+0000 7f81da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:28.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.421+0000 7f81ebac1640 5 monclient: authenticate success, global_id 4580 2024-03-29T10:48:28.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.421+0000 7f81ebac1640 10 monclient: _renew_subs 2024-03-29T10:48:28.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.421+0000 7f81ebac1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:28.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.421+0000 7f81ebac1640 1 -- 172.21.15.57:0/4216589193 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e406f920 con 0x7f81e41457f0 2024-03-29T10:48:28.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.421+0000 7f81ebac1640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:28.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.421+0000 7f81ebac1640 10 monclient: _renew_subs 2024-03-29T10:48:28.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.421+0000 7f81ebac1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:28.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.421+0000 7f81ebac1640 1 -- 172.21.15.57:0/4216589193 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e406dfb0 con 0x7f81e41457f0 2024-03-29T10:48:28.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.421+0000 7f81ebac1640 1 librados: init done 2024-03-29T10:48:28.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.422+0000 7f81ebac1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:28.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.422+0000 7f81ebac1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:28.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.422+0000 7f81ebac1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:28.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.422+0000 7f81ebac1640 1 -- 172.21.15.57:0/4216589193 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e4075f10 con 0x7f81e41457f0 2024-03-29T10:48:28.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.422+0000 7f81da7fc640 1 -- 172.21.15.57:0/4216589193 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81dc002030 con 0x7f81e41457f0 2024-03-29T10:48:28.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.423+0000 7f81da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:28.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.423+0000 7f81da7fc640 1 -- 172.21.15.57:0/4216589193 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f81dc003de0 con 0x7f81e41457f0 2024-03-29T10:48:28.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.423+0000 7f81da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:28.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.423+0000 7f81da7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:28.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.423+0000 7f81da7fc640 10 monclient: dump: 2024-03-29T10:48:28.414 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:28.414 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:28.414 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:28.414 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:28.414 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:28.415 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:28.415 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:28.415 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:28.415 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:28.415 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:28.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.423+0000 7f81db7fe640 4 set_mon_vals no callback set 2024-03-29T10:48:28.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.424+0000 7f81da7fc640 1 -- 172.21.15.57:0/4216589193 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f81dc003f40 con 0x7f81e41457f0 2024-03-29T10:48:28.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.425+0000 7f81e985d640 10 monclient: get_auth_request con 0x7f81c806dc90 auth_method 0 2024-03-29T10:48:28.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.426+0000 7f81da7fc640 1 -- 172.21.15.57:0/4216589193 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f81dc084ea0 con 0x7f81e41457f0 2024-03-29T10:48:28.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.426+0000 7f81da7fc640 10 client.4580.objecter ms_dispatch 0x7f81e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:28.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.426+0000 7f81da7fc640 3 client.4580.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:28.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.426+0000 7f81da7fc640 3 client.4580.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:28.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.426+0000 7f81da7fc640 20 client.4580.objecter dump_active .. 0 homeless 2024-03-29T10:48:28.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.426+0000 7f81da7fc640 10 client.4580.objecter ms_handle_connect 0x7f81c806dc90 2024-03-29T10:48:28.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.431+0000 7f81da7fc640 1 -- 172.21.15.57:0/4216589193 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f81dc050780 con 0x7f81e41457f0 2024-03-29T10:48:28.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.431+0000 7f81da7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:28.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.431+0000 7f81da7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:28.735 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.753+0000 7f81ebac1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:28.735 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.753+0000 7f81ebac1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:28.736 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.753+0000 7f81ebac1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:28.736 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.753+0000 7f81ebac1640 1 -- 172.21.15.57:0/4216589193 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81e40b3680 con 0x7f81e41457f0 2024-03-29T10:48:28.736 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.754+0000 7f81da7fc640 1 -- 172.21.15.57:0/4216589193 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f81dc0526b0 con 0x7f81e41457f0 2024-03-29T10:48:28.736 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.754+0000 7f81da7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:28.736 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.754+0000 7f81da7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:28.736 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:48:28.737 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:28.740 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.757+0000 7f81ebac1640 10 librados: watch_flush enter 2024-03-29T10:48:28.740 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.758+0000 7f81ebac1640 10 librados: watch_flush exit 2024-03-29T10:48:28.740 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.758+0000 7f81ebac1640 20 client.4580.objecter shutdown clearing up homeless session... 2024-03-29T10:48:28.740 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.758+0000 7f81ebac1640 10 client.4580.objecter successfully canceled tick 2024-03-29T10:48:28.740 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.758+0000 7f81ebac1640 1 -- 172.21.15.57:0/4216589193 >> v1:172.21.15.179:6816/2698295936 conn(0x7f81c806dc90 legacy=0x7f81c8070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:28.740 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.758+0000 7f81ebac1640 10 monclient: shutdown 2024-03-29T10:48:28.740 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.758+0000 7f81ebac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:28.740 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.758+0000 7f81ebac1640 1 -- 172.21.15.57:0/4216589193 >> v1:172.21.15.57:6790/0 conn(0x7f81e41457f0 legacy=0x7f81e4137ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:28.740 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.758+0000 7f81ebac1640 1 -- 172.21.15.57:0/4216589193 shutdown_connections 2024-03-29T10:48:28.740 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.758+0000 7f81ebac1640 1 -- 172.21.15.57:0/4216589193 >> 172.21.15.57:0/4216589193 conn(0x7f81e4066a00 msgr2=0x7f81e413b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:28.741 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.759+0000 7f81ebac1640 1 -- 172.21.15.57:0/4216589193 shutdown_connections 2024-03-29T10:48:28.741 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.759+0000 7f81ebac1640 1 -- 172.21.15.57:0/4216589193 wait complete. 2024-03-29T10:48:28.741 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:28.759+0000 7f81ebac1640 1 librados: shutdown 2024-03-29T10:48:28.754 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:35.560 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:35.694 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.712+0000 7fa947ac1640 10 monclient: get_monmap_and_config 2024-03-29T10:48:35.695 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.712+0000 7fa947ac1640 10 monclient: build_initial_monmap 2024-03-29T10:48:35.695 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.712+0000 7fa947ac1640 10 monclient: monmap: 2024-03-29T10:48:35.695 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:48:35.695 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:35.695 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:48:35.713267+0000 2024-03-29T10:48:35.695 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:48:35.713267+0000 2024-03-29T10:48:35.695 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:35.695 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:35.695 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:48:35.695 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:48:35.695 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:48:35.695 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:35.698 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.713+0000 7fa947ac1640 1 Processor -- start 2024-03-29T10:48:35.698 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.713+0000 7fa947ac1640 1 -- start start 2024-03-29T10:48:35.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.713+0000 7fa947ac1640 10 monclient: init 2024-03-29T10:48:35.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.714+0000 7fa947ac1640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:35.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.714+0000 7fa947ac1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:48:35.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.714+0000 7fa947ac1640 10 monclient(hunting): picked mon.noname-c con 0x7fa9401457f0 addr v1:172.21.15.179:6789/0 2024-03-29T10:48:35.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.714+0000 7fa947ac1640 10 monclient(hunting): picked mon.noname-a con 0x7fa94013f650 addr v1:172.21.15.57:6789/0 2024-03-29T10:48:35.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.714+0000 7fa947ac1640 10 monclient(hunting): picked mon.noname-b con 0x7fa9400a9d70 addr v1:172.21.15.57:6790/0 2024-03-29T10:48:35.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.715+0000 7fa947ac1640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9400aaef0 con 0x7fa94013f650 2024-03-29T10:48:35.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.715+0000 7fa947ac1640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa940145bd0 con 0x7fa9401457f0 2024-03-29T10:48:35.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.715+0000 7fa94585d640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fa9401457f0 0x7fa94013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:53908/0 (socket says 172.21.15.57:53908) 2024-03-29T10:48:35.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.715+0000 7fa94585d640 1 -- 172.21.15.57:0/3965322609 learned_addr learned my addr 172.21.15.57:0/3965322609 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:48:35.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.715+0000 7fa947ac1640 1 -- 172.21.15.57:0/3965322609 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa940145eb0 con 0x7fa9400a9d70 2024-03-29T10:48:35.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.715+0000 7fa947ac1640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:35.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.715+0000 7fa94485b640 1 -- 172.21.15.57:0/3965322609 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406146746 0 0) 0x7fa940145bd0 con 0x7fa9401457f0 2024-03-29T10:48:35.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.715+0000 7fa947ac1640 10 monclient(hunting): authenticate will time out at 1467.891716s 2024-03-29T10:48:35.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.715+0000 7fa94485b640 10 monclient(hunting): _negotiate 2024-03-29T10:48:35.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.715+0000 7fa94485b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:35.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.715+0000 7fa94485b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:35.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.716+0000 7fa94485b640 10 monclient(hunting): my global_id is 4531 2024-03-29T10:48:35.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.716+0000 7fa94485b640 1 -- 172.21.15.57:0/3965322609 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa92c003520 con 0x7fa9401457f0 2024-03-29T10:48:35.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.716+0000 7fa94485b640 1 -- 172.21.15.57:0/3965322609 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1577294346 0 0) 0x7fa940145eb0 con 0x7fa9400a9d70 2024-03-29T10:48:35.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.716+0000 7fa94485b640 10 monclient(hunting): _negotiate 2024-03-29T10:48:35.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.716+0000 7fa94485b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:35.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.716+0000 7fa94485b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:35.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.716+0000 7fa94485b640 10 monclient(hunting): my global_id is 4583 2024-03-29T10:48:35.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.716+0000 7fa94485b640 1 -- 172.21.15.57:0/3965322609 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa940145bd0 con 0x7fa9400a9d70 2024-03-29T10:48:35.701 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.716+0000 7fa94485b640 1 -- 172.21.15.57:0/3965322609 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2010048414 0 0) 0x7fa940145bd0 con 0x7fa9400a9d70 2024-03-29T10:48:35.701 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.716+0000 7fa94485b640 1 -- 172.21.15.57:0/3965322609 >> v1:172.21.15.179:6789/0 conn(0x7fa9401457f0 legacy=0x7fa94013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:35.701 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.716+0000 7fa94485b640 1 -- 172.21.15.57:0/3965322609 >> v1:172.21.15.57:6789/0 conn(0x7fa94013f650 legacy=0x7fa9400b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:48:35.703 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.718+0000 7fa94485b640 10 monclient: _finish_hunting 0 2024-03-29T10:48:35.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.718+0000 7fa94485b640 1 monclient: found mon.noname-b 2024-03-29T10:48:35.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.718+0000 7fa94485b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:35.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.718+0000 7fa94485b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:48:35.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.718+0000 7fa94485b640 1 -- 172.21.15.57:0/3965322609 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa940146340 con 0x7fa9400a9d70 2024-03-29T10:48:35.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.718+0000 7fa94485b640 10 monclient: _finish_auth 0 2024-03-29T10:48:35.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.718+0000 7fa94485b640 10 monclient: _check_auth_tickets 2024-03-29T10:48:35.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.718+0000 7fa94485b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:35.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.718+0000 7fa947ac1640 5 monclient: authenticate success, global_id 4583 2024-03-29T10:48:35.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.718+0000 7fa94485b640 1 -- 172.21.15.57:0/3965322609 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fa9380019f0 con 0x7fa9400a9d70 2024-03-29T10:48:35.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.718+0000 7fa947ac1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:35.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.718+0000 7fa94485b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:35.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.718+0000 7fa94485b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:35.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.718+0000 7fa94485b640 10 monclient: dump: 2024-03-29T10:48:35.705 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:35.705 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:35.705 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:35.705 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:35.705 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:35.705 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:35.705 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:35.705 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:35.705 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:35.705 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:35.705 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.718+0000 7fa947ac1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:35.705 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.719+0000 7fa94485b640 1 -- 172.21.15.57:0/3965322609 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa938001c10 con 0x7fa9400a9d70 2024-03-29T10:48:35.705 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.719+0000 7fa94485b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:35.706 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.719+0000 7fa94485b640 1 -- 172.21.15.57:0/3965322609 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fa938001fa0 con 0x7fa9400a9d70 2024-03-29T10:48:35.706 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.719+0000 7fa94485b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:35.706 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.719+0000 7fa94485b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:35.706 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.719+0000 7fa94485b640 10 monclient: dump: 2024-03-29T10:48:35.706 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:35.706 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:35.706 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:35.706 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:35.706 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:35.706 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:35.706 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:35.707 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:35.707 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:35.707 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:35.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.719+0000 7fa947ac1640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:35.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.719+0000 7fa947ac1640 4 set_mon_vals no callback set 2024-03-29T10:48:35.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.720+0000 7fa947ac1640 10 monclient: shutdown 2024-03-29T10:48:35.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.720+0000 7fa947ac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:35.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.720+0000 7fa947ac1640 1 -- 172.21.15.57:0/3965322609 >> v1:172.21.15.57:6790/0 conn(0x7fa9400a9d70 legacy=0x7fa94014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:35.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.720+0000 7fa947ac1640 1 -- 172.21.15.57:0/3965322609 shutdown_connections 2024-03-29T10:48:35.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.720+0000 7fa947ac1640 1 -- 172.21.15.57:0/3965322609 >> 172.21.15.57:0/3965322609 conn(0x7fa940066a00 msgr2=0x7fa940066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:35.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.722+0000 7fa947ac1640 1 -- 172.21.15.57:0/3965322609 shutdown_connections 2024-03-29T10:48:35.714 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.722+0000 7fa947ac1640 1 -- 172.21.15.57:0/3965322609 wait complete. 2024-03-29T10:48:35.714 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.723+0000 7fa947ac1640 10 monclient: build_initial_monmap 2024-03-29T10:48:35.714 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.723+0000 7fa947ac1640 10 monclient: monmap: 2024-03-29T10:48:35.714 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:48:35.714 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:35.714 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:48:35.714 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:48:35.714 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:35.715 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:35.715 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:48:35.715 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:48:35.715 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:48:35.715 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:35.715 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.724+0000 7fa947ac1640 1 librados: starting msgr at 2024-03-29T10:48:35.715 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.724+0000 7fa947ac1640 1 librados: starting objecter 2024-03-29T10:48:35.715 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.724+0000 7fa947ac1640 1 Processor -- start 2024-03-29T10:48:35.715 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.724+0000 7fa947ac1640 1 -- start start 2024-03-29T10:48:35.715 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.724+0000 7fa947ac1640 1 librados: setting wanted keys 2024-03-29T10:48:35.715 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.724+0000 7fa947ac1640 1 librados: calling monclient init 2024-03-29T10:48:35.715 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.724+0000 7fa947ac1640 10 monclient: init 2024-03-29T10:48:35.715 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.725+0000 7fa947ac1640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:35.715 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.725+0000 7fa947ac1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:48:35.715 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.725+0000 7fa947ac1640 10 monclient(hunting): picked mon.noname-b con 0x7fa9400a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T10:48:35.716 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.725+0000 7fa947ac1640 10 monclient(hunting): picked mon.noname-c con 0x7fa94013f650 addr v1:172.21.15.57:6790/0 2024-03-29T10:48:35.716 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.725+0000 7fa947ac1640 10 monclient(hunting): picked mon.noname-a con 0x7fa9401457f0 addr v1:172.21.15.57:6789/0 2024-03-29T10:48:35.716 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.725+0000 7fa94505c640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7fa94013f650 0x7fa940136da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:36768/0 (socket says 172.21.15.57:36768) 2024-03-29T10:48:35.716 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.725+0000 7fa94505c640 1 -- 172.21.15.57:0/3298217456 learned_addr learned my addr 172.21.15.57:0/3298217456 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:48:35.716 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.725+0000 7fa947ac1640 1 -- 172.21.15.57:0/3298217456 --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa940137bc0 con 0x7fa9401457f0 2024-03-29T10:48:35.716 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.725+0000 7fa947ac1640 1 -- 172.21.15.57:0/3298217456 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9400ac4b0 con 0x7fa9400a9d70 2024-03-29T10:48:35.716 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.725+0000 7fa947ac1640 1 -- 172.21.15.57:0/3298217456 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa940142df0 con 0x7fa94013f650 2024-03-29T10:48:35.716 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.725+0000 7fa947ac1640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:35.716 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.725+0000 7fa947ac1640 10 monclient(hunting): authenticate will time out at 1467.901900s 2024-03-29T10:48:35.716 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.726+0000 7fa9367fc640 10 client.?.objecter ms_handle_connect 0x7fa94013f650 2024-03-29T10:48:35.716 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.726+0000 7fa9367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:35.716 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.726+0000 7fa9367fc640 10 client.?.objecter ms_handle_connect 0x7fa9400a9d70 2024-03-29T10:48:35.716 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.726+0000 7fa9367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:35.716 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.726+0000 7fa9367fc640 1 -- 172.21.15.57:0/3298217456 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011263856 0 0) 0x7fa940142df0 con 0x7fa94013f650 2024-03-29T10:48:35.717 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.726+0000 7fa9367fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:35.717 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.726+0000 7fa9367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:35.717 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.726+0000 7fa9367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:35.717 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.726+0000 7fa9367fc640 10 monclient(hunting): my global_id is 4586 2024-03-29T10:48:35.717 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.726+0000 7fa9367fc640 1 -- 172.21.15.57:0/3298217456 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa924004eb0 con 0x7fa94013f650 2024-03-29T10:48:35.717 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.726+0000 7fa9367fc640 1 -- 172.21.15.57:0/3298217456 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551876266 0 0) 0x7fa9400ac4b0 con 0x7fa9400a9d70 2024-03-29T10:48:35.717 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.726+0000 7fa9367fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:35.717 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.726+0000 7fa9367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:35.717 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.726+0000 7fa9367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:35.717 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.726+0000 7fa9367fc640 10 monclient(hunting): my global_id is 4534 2024-03-29T10:48:35.717 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.726+0000 7fa9367fc640 1 -- 172.21.15.57:0/3298217456 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa940142df0 con 0x7fa9400a9d70 2024-03-29T10:48:35.717 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.728+0000 7fa9367fc640 1 -- 172.21.15.57:0/3298217456 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 10839557 0 0) 0x7fa924004eb0 con 0x7fa94013f650 2024-03-29T10:48:35.717 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.728+0000 7fa9367fc640 1 -- 172.21.15.57:0/3298217456 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9400ac4b0 con 0x7fa94013f650 2024-03-29T10:48:35.717 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.728+0000 7fa9367fc640 1 -- 172.21.15.57:0/3298217456 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fa930001960 con 0x7fa94013f650 2024-03-29T10:48:35.718 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.728+0000 7fa9367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:35.718 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.728+0000 7fa9367fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:35.718 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.728+0000 7fa9367fc640 10 monclient(hunting): dump: 2024-03-29T10:48:35.718 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:35.718 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:35.718 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:35.718 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:35.718 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:35.718 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:35.718 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:35.718 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:35.718 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:35.718 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:35.718 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.728+0000 7fa9367fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:35.719 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.729+0000 7fa9367fc640 10 client.?.objecter ms_handle_connect 0x7fa9401457f0 2024-03-29T10:48:35.719 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.729+0000 7fa9367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:35.719 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.729+0000 7fa9367fc640 1 -- 172.21.15.57:0/3298217456 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3306288308 0 0) 0x7fa9400ac4b0 con 0x7fa94013f650 2024-03-29T10:48:35.719 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.729+0000 7fa9367fc640 1 -- 172.21.15.57:0/3298217456 >> v1:172.21.15.179:6789/0 conn(0x7fa9400a9d70 legacy=0x7fa9401426e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:35.719 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.730+0000 7fa9367fc640 1 -- 172.21.15.57:0/3298217456 >> v1:172.21.15.57:6789/0 conn(0x7fa9401457f0 legacy=0x7fa9401374b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:35.719 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.730+0000 7fa9367fc640 10 monclient: _finish_hunting 0 2024-03-29T10:48:35.719 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.730+0000 7fa9367fc640 1 monclient: found mon.c 2024-03-29T10:48:35.719 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.730+0000 7fa9367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:35.719 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.730+0000 7fa9367fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:35.719 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.730+0000 7fa9367fc640 1 -- 172.21.15.57:0/3298217456 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9400ab600 con 0x7fa94013f650 2024-03-29T10:48:35.719 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.730+0000 7fa9367fc640 10 monclient: _finish_auth 0 2024-03-29T10:48:35.719 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.730+0000 7fa9367fc640 10 monclient: _check_auth_tickets 2024-03-29T10:48:35.719 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.730+0000 7fa9367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:35.719 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.730+0000 7fa947ac1640 5 monclient: authenticate success, global_id 4586 2024-03-29T10:48:35.719 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.730+0000 7fa947ac1640 10 monclient: _renew_subs 2024-03-29T10:48:35.720 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.730+0000 7fa947ac1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:35.720 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.730+0000 7fa947ac1640 1 -- 172.21.15.57:0/3298217456 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa940141db0 con 0x7fa94013f650 2024-03-29T10:48:35.720 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.731+0000 7fa947ac1640 10 client.4586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:35.720 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.731+0000 7fa947ac1640 10 monclient: _renew_subs 2024-03-29T10:48:35.720 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.731+0000 7fa947ac1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:35.720 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.731+0000 7fa947ac1640 1 -- 172.21.15.57:0/3298217456 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa94013ff80 con 0x7fa94013f650 2024-03-29T10:48:35.720 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.731+0000 7fa947ac1640 1 librados: init done 2024-03-29T10:48:35.720 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.732+0000 7fa9367fc640 1 -- 172.21.15.57:0/3298217456 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9300035f0 con 0x7fa94013f650 2024-03-29T10:48:35.720 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.732+0000 7fa9367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:35.720 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.732+0000 7fa9367fc640 1 -- 172.21.15.57:0/3298217456 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fa930003940 con 0x7fa94013f650 2024-03-29T10:48:35.720 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.732+0000 7fa9367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:35.720 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.732+0000 7fa9367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:35.720 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.732+0000 7fa9367fc640 10 monclient: dump: 2024-03-29T10:48:35.720 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:35.721 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:35.721 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:35.721 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:35.721 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:35.721 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:35.721 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:35.721 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:35.721 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:35.721 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:35.721 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.732+0000 7fa937fff640 4 set_mon_vals no callback set 2024-03-29T10:48:35.723 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.732+0000 7fa947ac1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:35.723 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.733+0000 7fa947ac1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:35.723 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.733+0000 7fa947ac1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:35.723 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.733+0000 7fa947ac1640 1 -- 172.21.15.57:0/3298217456 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa928005080 con 0x7fa94013f650 2024-03-29T10:48:35.723 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.734+0000 7fa9367fc640 1 -- 172.21.15.57:0/3298217456 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fa930003aa0 con 0x7fa94013f650 2024-03-29T10:48:35.723 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.734+0000 7fa9367fc640 1 -- 172.21.15.57:0/3298217456 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7fa930084a60 con 0x7fa94013f650 2024-03-29T10:48:35.723 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.734+0000 7fa9367fc640 10 client.4586.objecter ms_dispatch 0x7fa940000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:35.723 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.734+0000 7fa9367fc640 3 client.4586.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:35.723 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.734+0000 7fa9367fc640 3 client.4586.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:35.723 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.734+0000 7fa9367fc640 20 client.4586.objecter dump_active .. 0 homeless 2024-03-29T10:48:35.724 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.735+0000 7fa94585d640 10 monclient: get_auth_request con 0x7fa92406ddb0 auth_method 0 2024-03-29T10:48:35.724 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.735+0000 7fa9367fc640 10 client.4586.objecter ms_handle_connect 0x7fa92406ddb0 2024-03-29T10:48:35.724 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.740+0000 7fa9367fc640 1 -- 172.21.15.57:0/3298217456 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa930050340 con 0x7fa94013f650 2024-03-29T10:48:35.724 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.740+0000 7fa9367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:35.724 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:35.740+0000 7fa9367fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:36.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:36.065+0000 7fa947ac1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:36.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:36.065+0000 7fa947ac1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:36.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:36.065+0000 7fa947ac1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:36.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:36.065+0000 7fa947ac1640 1 -- 172.21.15.57:0/3298217456 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa928005330 con 0x7fa94013f650 2024-03-29T10:48:36.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:36.067+0000 7fa9367fc640 1 -- 172.21.15.57:0/3298217456 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7fa930052270 con 0x7fa94013f650 2024-03-29T10:48:36.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:36.067+0000 7fa9367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:36.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:36.067+0000 7fa9367fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:36.049 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:48:36.049 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:36.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:36.070+0000 7fa947ac1640 10 librados: watch_flush enter 2024-03-29T10:48:36.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:36.070+0000 7fa947ac1640 10 librados: watch_flush exit 2024-03-29T10:48:36.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:36.070+0000 7fa947ac1640 20 client.4586.objecter shutdown clearing up homeless session... 2024-03-29T10:48:36.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:36.070+0000 7fa947ac1640 10 client.4586.objecter successfully canceled tick 2024-03-29T10:48:36.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:36.070+0000 7fa947ac1640 1 -- 172.21.15.57:0/3298217456 >> v1:172.21.15.179:6816/2698295936 conn(0x7fa92406ddb0 legacy=0x7fa924070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:36.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:36.070+0000 7fa947ac1640 10 monclient: shutdown 2024-03-29T10:48:36.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:36.070+0000 7fa947ac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:36.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:36.070+0000 7fa947ac1640 1 -- 172.21.15.57:0/3298217456 >> v1:172.21.15.57:6790/0 conn(0x7fa94013f650 legacy=0x7fa940136da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:36.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:36.070+0000 7fa947ac1640 1 -- 172.21.15.57:0/3298217456 shutdown_connections 2024-03-29T10:48:36.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:36.070+0000 7fa947ac1640 1 -- 172.21.15.57:0/3298217456 >> 172.21.15.57:0/3298217456 conn(0x7fa940066a00 msgr2=0x7fa940066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:36.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:36.071+0000 7fa947ac1640 1 -- 172.21.15.57:0/3298217456 shutdown_connections 2024-03-29T10:48:36.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:36.071+0000 7fa947ac1640 1 -- 172.21.15.57:0/3298217456 wait complete. 2024-03-29T10:48:36.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:36.071+0000 7fa947ac1640 1 librados: shutdown 2024-03-29T10:48:36.068 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:42.875 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:43.025 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.042+0000 7f16113fe640 10 monclient: get_monmap_and_config 2024-03-29T10:48:43.025 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.042+0000 7f16113fe640 10 monclient: build_initial_monmap 2024-03-29T10:48:43.025 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.042+0000 7f16113fe640 10 monclient: monmap: 2024-03-29T10:48:43.025 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:48:43.026 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:43.026 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:48:43.042928+0000 2024-03-29T10:48:43.026 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:48:43.042928+0000 2024-03-29T10:48:43.026 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:43.026 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:43.026 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:48:43.026 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:48:43.027 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:48:43.027 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:43.027 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.043+0000 7f16113fe640 1 Processor -- start 2024-03-29T10:48:43.027 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.044+0000 7f16113fe640 1 -- start start 2024-03-29T10:48:43.027 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.044+0000 7f16113fe640 10 monclient: init 2024-03-29T10:48:43.029 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.045+0000 7f16113fe640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:43.029 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.045+0000 7f16113fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:48:43.030 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.045+0000 7f16113fe640 10 monclient(hunting): picked mon.noname-b con 0x7f160c0ad6c0 addr v1:172.21.15.57:6790/0 2024-03-29T10:48:43.030 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.045+0000 7f16113fe640 10 monclient(hunting): picked mon.noname-a con 0x7f160c142db0 addr v1:172.21.15.57:6789/0 2024-03-29T10:48:43.030 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.045+0000 7f16113fe640 10 monclient(hunting): picked mon.noname-c con 0x7f160c14ba40 addr v1:172.21.15.179:6789/0 2024-03-29T10:48:43.030 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.045+0000 7f16113fe640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160c0adaa0 con 0x7f160c142db0 2024-03-29T10:48:43.030 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.045+0000 7f16113fe640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160c0aab80 con 0x7f160c14ba40 2024-03-29T10:48:43.030 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.045+0000 7f16113fe640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160c1425f0 con 0x7f160c0ad6c0 2024-03-29T10:48:43.030 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.045+0000 7f16113fe640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:43.031 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.045+0000 7f16113fe640 10 monclient(hunting): authenticate will time out at 1475.221306s 2024-03-29T10:48:43.034 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f160a7fc640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f160c142db0 0x7f160c0ac7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:59643/0 (socket says 172.21.15.57:59643) 2024-03-29T10:48:43.034 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f160a7fc640 1 -- 172.21.15.57:0/319771965 learned_addr learned my addr 172.21.15.57:0/319771965 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:48:43.034 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f1609ffb640 1 -- 172.21.15.57:0/319771965 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163605623 0 0) 0x7f160c0adaa0 con 0x7f160c142db0 2024-03-29T10:48:43.034 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f1609ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:48:43.034 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f1609ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:43.035 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f1609ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:43.035 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f1609ffb640 10 monclient(hunting): my global_id is 4533 2024-03-29T10:48:43.035 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f1609ffb640 1 -- 172.21.15.57:0/319771965 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f4003520 con 0x7f160c142db0 2024-03-29T10:48:43.035 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f1609ffb640 1 -- 172.21.15.57:0/319771965 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 894439954 0 0) 0x7f15f4003520 con 0x7f160c142db0 2024-03-29T10:48:43.035 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f1609ffb640 1 -- 172.21.15.57:0/319771965 >> v1:172.21.15.57:6790/0 conn(0x7f160c0ad6c0 legacy=0x7f160c1456b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:43.035 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f1609ffb640 1 -- 172.21.15.57:0/319771965 >> v1:172.21.15.179:6789/0 conn(0x7f160c14ba40 legacy=0x7f160c14de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:48:43.035 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f1609ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:48:43.036 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f1609ffb640 1 monclient: found mon.noname-a 2024-03-29T10:48:43.036 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f1609ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:43.036 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f1609ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:48:43.036 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f1609ffb640 1 -- 172.21.15.57:0/319771965 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160c0aae60 con 0x7f160c142db0 2024-03-29T10:48:43.036 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f1609ffb640 10 monclient: _finish_auth 0 2024-03-29T10:48:43.036 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f1609ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:48:43.036 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f1609ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:43.036 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f16113fe640 5 monclient: authenticate success, global_id 4533 2024-03-29T10:48:43.036 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f16113fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:43.036 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f1609ffb640 1 -- 172.21.15.57:0/319771965 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1600001690 con 0x7f160c142db0 2024-03-29T10:48:43.037 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f1609ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:43.037 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f1609ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:48:43.037 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.049+0000 7f1609ffb640 10 monclient: dump: 2024-03-29T10:48:43.037 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:43.037 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:43.037 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:43.037 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:43.038 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:43.038 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:43.038 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:43.038 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:43.038 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:43.038 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:43.038 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.050+0000 7f16113fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:43.038 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.050+0000 7f1609ffb640 1 -- 172.21.15.57:0/319771965 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1600001920 con 0x7f160c142db0 2024-03-29T10:48:43.039 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.050+0000 7f1609ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:43.039 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.050+0000 7f16113fe640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:43.039 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.050+0000 7f16113fe640 4 set_mon_vals no callback set 2024-03-29T10:48:43.039 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.051+0000 7f16113fe640 10 monclient: shutdown 2024-03-29T10:48:43.039 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.051+0000 7f16113fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:43.039 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.051+0000 7f16113fe640 1 -- 172.21.15.57:0/319771965 >> v1:172.21.15.57:6789/0 conn(0x7f160c142db0 legacy=0x7f160c0ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:43.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.052+0000 7f16113fe640 1 -- 172.21.15.57:0/319771965 shutdown_connections 2024-03-29T10:48:43.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.052+0000 7f16113fe640 1 -- 172.21.15.57:0/319771965 >> 172.21.15.57:0/319771965 conn(0x7f160c0b35e0 msgr2=0x7f160c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:43.044 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.052+0000 7f16113fe640 1 -- 172.21.15.57:0/319771965 shutdown_connections 2024-03-29T10:48:43.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.052+0000 7f16113fe640 1 -- 172.21.15.57:0/319771965 wait complete. 2024-03-29T10:48:43.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.053+0000 7f16113fe640 10 monclient: build_initial_monmap 2024-03-29T10:48:43.045 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.053+0000 7f16113fe640 10 monclient: monmap: 2024-03-29T10:48:43.045 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:48:43.045 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:43.045 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:48:43.045 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:48:43.045 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:43.045 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:43.045 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:48:43.046 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:48:43.046 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:48:43.046 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:43.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.054+0000 7f16113fe640 1 librados: starting msgr at 2024-03-29T10:48:43.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.054+0000 7f16113fe640 1 librados: starting objecter 2024-03-29T10:48:43.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.054+0000 7f16113fe640 1 Processor -- start 2024-03-29T10:48:43.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.054+0000 7f16113fe640 1 -- start start 2024-03-29T10:48:43.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.054+0000 7f16113fe640 1 librados: setting wanted keys 2024-03-29T10:48:43.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.054+0000 7f16113fe640 1 librados: calling monclient init 2024-03-29T10:48:43.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.054+0000 7f16113fe640 10 monclient: init 2024-03-29T10:48:43.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.055+0000 7f16113fe640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:43.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.055+0000 7f16113fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:48:43.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.055+0000 7f16113fe640 10 monclient(hunting): picked mon.noname-c con 0x7f160c0ad6c0 addr v1:172.21.15.57:6790/0 2024-03-29T10:48:43.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.055+0000 7f16113fe640 10 monclient(hunting): picked mon.noname-b con 0x7f160c142db0 addr v1:172.21.15.179:6789/0 2024-03-29T10:48:43.046 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.055+0000 7f16113fe640 10 monclient(hunting): picked mon.noname-a con 0x7f160c14ba40 addr v1:172.21.15.57:6789/0 2024-03-29T10:48:43.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.055+0000 7f16113fe640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160c1442d0 con 0x7f160c14ba40 2024-03-29T10:48:43.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.055+0000 7f16113fe640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160c06f4b0 con 0x7f160c142db0 2024-03-29T10:48:43.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.055+0000 7f16113fe640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160c06f790 con 0x7f160c0ad6c0 2024-03-29T10:48:43.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.055+0000 7f16113fe640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:43.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.055+0000 7f16113fe640 10 monclient(hunting): authenticate will time out at 1475.231653s 2024-03-29T10:48:43.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.056+0000 7f160affd640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f160c0ad6c0 0x7f160c070170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:46256/0 (socket says 172.21.15.57:46256) 2024-03-29T10:48:43.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.056+0000 7f160affd640 1 -- 172.21.15.57:0/3956343470 learned_addr learned my addr 172.21.15.57:0/3956343470 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:48:43.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.058+0000 7f15f37fe640 10 client.?.objecter ms_handle_connect 0x7f160c0ad6c0 2024-03-29T10:48:43.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.058+0000 7f15f37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:43.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.058+0000 7f15f37fe640 1 -- 172.21.15.57:0/3956343470 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2627367066 0 0) 0x7f160c06f790 con 0x7f160c0ad6c0 2024-03-29T10:48:43.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.058+0000 7f15f37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:43.047 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.058+0000 7f15f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:43.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.058+0000 7f15f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:43.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.058+0000 7f15f37fe640 10 monclient(hunting): my global_id is 4589 2024-03-29T10:48:43.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.058+0000 7f15f37fe640 1 -- 172.21.15.57:0/3956343470 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f8004eb0 con 0x7f160c0ad6c0 2024-03-29T10:48:43.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.058+0000 7f15f37fe640 1 -- 172.21.15.57:0/3956343470 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3300308908 0 0) 0x7f15f8004eb0 con 0x7f160c0ad6c0 2024-03-29T10:48:43.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.058+0000 7f15f37fe640 1 -- 172.21.15.57:0/3956343470 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f160c06f790 con 0x7f160c0ad6c0 2024-03-29T10:48:43.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.058+0000 7f15f37fe640 1 -- 172.21.15.57:0/3956343470 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1604004400 con 0x7f160c0ad6c0 2024-03-29T10:48:43.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.058+0000 7f15f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:43.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.058+0000 7f15f37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:43.048 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.058+0000 7f15f37fe640 10 monclient(hunting): dump: 2024-03-29T10:48:43.048 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:43.048 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:43.048 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:43.049 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:43.049 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:43.049 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:43.049 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:43.049 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:43.049 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:43.049 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:43.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.059+0000 7f15f37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:43.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.060+0000 7f15f37fe640 10 client.?.objecter ms_handle_connect 0x7f160c14ba40 2024-03-29T10:48:43.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.060+0000 7f15f37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:43.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.060+0000 7f160b7fe640 1 -- 172.21.15.57:0/3956343470 >> v1:172.21.15.57:6789/0 conn(0x7f160c14ba40 legacy=0x7f160c143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-29T10:48:43.049 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.060+0000 7f160b7fe640 1 -- 172.21.15.57:0/3956343470 >> v1:172.21.15.57:6789/0 conn(0x7f160c14ba40 legacy=0x7f160c143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:48:43.050 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.060+0000 7f160b7fe640 1 --1- 172.21.15.57:0/3956343470 >> v1:172.21.15.57:6789/0 conn(0x7f160c14ba40 0x7f160c143bc0 :-1 s=OPENED pgs=877326 cs=1 l=1).handle_message read tag failed 2024-03-29T10:48:43.050 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.060+0000 7f160b7fe640 1 --1- 172.21.15.57:0/3956343470 >> v1:172.21.15.57:6789/0 conn(0x7f160c14ba40 0x7f160c143bc0 :-1 s=OPENED pgs=877326 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:48:43.050 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.060+0000 7f15f37fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.57:6789/0 2024-03-29T10:48:43.050 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.060+0000 7f15f37fe640 1 -- 172.21.15.57:0/3956343470 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3246610910 0 0) 0x7f160c06f790 con 0x7f160c0ad6c0 2024-03-29T10:48:43.050 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.060+0000 7f15f37fe640 1 -- 172.21.15.57:0/3956343470 >> v1:172.21.15.179:6789/0 conn(0x7f160c142db0 legacy=0x7f160c1434b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:48:43.050 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.062+0000 7f15f37fe640 1 -- 172.21.15.57:0/3956343470 >> v1:172.21.15.57:6789/0 conn(0x7f160c14ba40 legacy=0x7f160c143bc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:48:43.050 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.062+0000 7f15f37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:48:43.050 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.062+0000 7f15f37fe640 1 monclient: found mon.c 2024-03-29T10:48:43.050 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.062+0000 7f15f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:43.050 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.062+0000 7f15f37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:43.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.062+0000 7f15f37fe640 1 -- 172.21.15.57:0/3956343470 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160c06fa70 con 0x7f160c0ad6c0 2024-03-29T10:48:43.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.062+0000 7f15f37fe640 10 monclient: _finish_auth 0 2024-03-29T10:48:43.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.062+0000 7f15f37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:48:43.052 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.062+0000 7f15f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:43.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.062+0000 7f16113fe640 5 monclient: authenticate success, global_id 4589 2024-03-29T10:48:43.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.062+0000 7f16113fe640 10 monclient: _renew_subs 2024-03-29T10:48:43.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.062+0000 7f16113fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:43.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.062+0000 7f16113fe640 1 -- 172.21.15.57:0/3956343470 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f160c13a350 con 0x7f160c0ad6c0 2024-03-29T10:48:43.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.062+0000 7f16113fe640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:43.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.062+0000 7f16113fe640 10 monclient: _renew_subs 2024-03-29T10:48:43.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.062+0000 7f16113fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:43.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.062+0000 7f16113fe640 1 -- 172.21.15.57:0/3956343470 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f160c06f0b0 con 0x7f160c0ad6c0 2024-03-29T10:48:43.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.062+0000 7f16113fe640 1 librados: init done 2024-03-29T10:48:43.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.063+0000 7f16113fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:43.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.063+0000 7f16113fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:43.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.063+0000 7f16113fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:43.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.063+0000 7f16113fe640 1 -- 172.21.15.57:0/3956343470 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f160c13a590 con 0x7f160c0ad6c0 2024-03-29T10:48:43.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.065+0000 7f15f37fe640 1 -- 172.21.15.57:0/3956343470 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1604004720 con 0x7f160c0ad6c0 2024-03-29T10:48:43.053 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.065+0000 7f15f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:43.054 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.065+0000 7f15f37fe640 1 -- 172.21.15.57:0/3956343470 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f1604004400 con 0x7f160c0ad6c0 2024-03-29T10:48:43.054 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.065+0000 7f15f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:43.054 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.065+0000 7f15f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:43.054 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.065+0000 7f15f37fe640 10 monclient: dump: 2024-03-29T10:48:43.054 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:43.054 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:43.054 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:43.054 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:43.054 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:43.054 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:43.054 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:43.054 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:43.055 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:43.055 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:43.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.065+0000 7f15f37fe640 1 -- 172.21.15.57:0/3956343470 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f1604001540 con 0x7f160c0ad6c0 2024-03-29T10:48:43.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.065+0000 7f16097fa640 4 set_mon_vals no callback set 2024-03-29T10:48:43.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.066+0000 7f15f37fe640 1 -- 172.21.15.57:0/3956343470 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f1604084c90 con 0x7f160c0ad6c0 2024-03-29T10:48:43.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.066+0000 7f15f37fe640 10 client.4589.objecter ms_dispatch 0x7f160c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:43.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.066+0000 7f15f37fe640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:43.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.066+0000 7f15f37fe640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:43.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.066+0000 7f15f37fe640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-29T10:48:43.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.067+0000 7f160a7fc640 10 monclient: get_auth_request con 0x7f15f806df10 auth_method 0 2024-03-29T10:48:43.055 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.069+0000 7f15f37fe640 10 client.4589.objecter ms_handle_connect 0x7f15f806df10 2024-03-29T10:48:43.056 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.072+0000 7f15f37fe640 1 -- 172.21.15.57:0/3956343470 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1604050420 con 0x7f160c0ad6c0 2024-03-29T10:48:43.056 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.073+0000 7f15f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:43.056 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.073+0000 7f15f37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:43.403 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.420+0000 7f16113fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:43.403 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.420+0000 7f16113fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:43.403 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.420+0000 7f16113fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:43.403 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.420+0000 7f16113fe640 1 -- 172.21.15.57:0/3956343470 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f160c075f10 con 0x7f160c0ad6c0 2024-03-29T10:48:43.404 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.422+0000 7f15f37fe640 1 -- 172.21.15.57:0/3956343470 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f1604052520 con 0x7f160c0ad6c0 2024-03-29T10:48:43.404 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.422+0000 7f15f37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:43.404 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.422+0000 7f15f37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:43.406 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:48:43.407 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:43.408 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.425+0000 7f16113fe640 10 librados: watch_flush enter 2024-03-29T10:48:43.408 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.425+0000 7f16113fe640 10 librados: watch_flush exit 2024-03-29T10:48:43.408 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.425+0000 7f16113fe640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-29T10:48:43.408 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.425+0000 7f16113fe640 10 client.4589.objecter successfully canceled tick 2024-03-29T10:48:43.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.425+0000 7f16113fe640 1 -- 172.21.15.57:0/3956343470 >> v1:172.21.15.179:6816/2698295936 conn(0x7f15f806df10 legacy=0x7f15f80703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:43.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.425+0000 7f16113fe640 10 monclient: shutdown 2024-03-29T10:48:43.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.425+0000 7f16113fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:43.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.425+0000 7f16113fe640 1 -- 172.21.15.57:0/3956343470 >> v1:172.21.15.57:6790/0 conn(0x7f160c0ad6c0 legacy=0x7f160c070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:43.409 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.425+0000 7f16113fe640 1 -- 172.21.15.57:0/3956343470 shutdown_connections 2024-03-29T10:48:43.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.425+0000 7f16113fe640 1 -- 172.21.15.57:0/3956343470 >> 172.21.15.57:0/3956343470 conn(0x7f160c0b35e0 msgr2=0x7f160c0adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:43.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.426+0000 7f16113fe640 1 -- 172.21.15.57:0/3956343470 shutdown_connections 2024-03-29T10:48:43.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.426+0000 7f16113fe640 1 -- 172.21.15.57:0/3956343470 wait complete. 2024-03-29T10:48:43.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:43.426+0000 7f16113fe640 1 librados: shutdown 2024-03-29T10:48:43.425 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:50.232 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:50.384 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.402+0000 7f23840b3640 10 monclient: get_monmap_and_config 2024-03-29T10:48:50.384 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.402+0000 7f23840b3640 10 monclient: build_initial_monmap 2024-03-29T10:48:50.384 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.402+0000 7f23840b3640 10 monclient: monmap: 2024-03-29T10:48:50.384 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:48:50.384 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:50.384 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:48:50.402991+0000 2024-03-29T10:48:50.384 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:48:50.402991+0000 2024-03-29T10:48:50.384 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:50.384 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:50.385 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:48:50.385 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:48:50.385 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:48:50.385 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:50.385 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.403+0000 7f23840b3640 1 Processor -- start 2024-03-29T10:48:50.385 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.404+0000 7f23840b3640 1 -- start start 2024-03-29T10:48:50.386 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.404+0000 7f23840b3640 10 monclient: init 2024-03-29T10:48:50.386 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.405+0000 7f23840b3640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:50.387 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.405+0000 7f23840b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:48:50.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.405+0000 7f23840b3640 10 monclient(hunting): picked mon.noname-c con 0x7f237c1457f0 addr v1:172.21.15.179:6789/0 2024-03-29T10:48:50.389 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.405+0000 7f23840b3640 10 monclient(hunting): picked mon.noname-b con 0x7f237c13f650 addr v1:172.21.15.57:6790/0 2024-03-29T10:48:50.391 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.409+0000 7f23840b3640 10 monclient(hunting): picked mon.noname-a con 0x7f237c0a9d70 addr v1:172.21.15.57:6789/0 2024-03-29T10:48:50.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.409+0000 7f238164e640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f237c13f650 0x7f237c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:35118/0 (socket says 172.21.15.57:35118) 2024-03-29T10:48:50.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.409+0000 7f238164e640 1 -- 172.21.15.57:0/2959735036 learned_addr learned my addr 172.21.15.57:0/2959735036 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:48:50.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.409+0000 7f23840b3640 1 -- 172.21.15.57:0/2959735036 --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c0aaef0 con 0x7f237c0a9d70 2024-03-29T10:48:50.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.409+0000 7f23840b3640 1 -- 172.21.15.57:0/2959735036 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c145bd0 con 0x7f237c1457f0 2024-03-29T10:48:50.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.409+0000 7f23840b3640 1 -- 172.21.15.57:0/2959735036 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c145eb0 con 0x7f237c13f650 2024-03-29T10:48:50.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.409+0000 7f23840b3640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:50.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.409+0000 7f23840b3640 10 monclient(hunting): authenticate will time out at 1482.585419s 2024-03-29T10:48:50.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.409+0000 7f2380e4d640 1 -- 172.21.15.57:0/2959735036 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752359387 0 0) 0x7f237c0aaef0 con 0x7f237c0a9d70 2024-03-29T10:48:50.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.409+0000 7f2380e4d640 10 monclient(hunting): _negotiate 2024-03-29T10:48:50.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.409+0000 7f2380e4d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:50.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.409+0000 7f2380e4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:50.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.409+0000 7f2380e4d640 10 monclient(hunting): my global_id is 4536 2024-03-29T10:48:50.392 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.409+0000 7f2380e4d640 1 -- 172.21.15.57:0/2959735036 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2368003520 con 0x7f237c0a9d70 2024-03-29T10:48:50.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.410+0000 7f2380e4d640 1 -- 172.21.15.57:0/2959735036 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3920125391 0 0) 0x7f237c145eb0 con 0x7f237c13f650 2024-03-29T10:48:50.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.410+0000 7f2380e4d640 10 monclient(hunting): _negotiate 2024-03-29T10:48:50.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.410+0000 7f2380e4d640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:50.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.410+0000 7f2380e4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:50.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.410+0000 7f2380e4d640 10 monclient(hunting): my global_id is 4592 2024-03-29T10:48:50.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.410+0000 7f2380e4d640 1 -- 172.21.15.57:0/2959735036 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f237c0aaef0 con 0x7f237c13f650 2024-03-29T10:48:50.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.411+0000 7f2380e4d640 1 -- 172.21.15.57:0/2959735036 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4104574191 0 0) 0x7f237c0aaef0 con 0x7f237c13f650 2024-03-29T10:48:50.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.411+0000 7f2380e4d640 1 -- 172.21.15.57:0/2959735036 >> v1:172.21.15.179:6789/0 conn(0x7f237c1457f0 legacy=0x7f237c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:50.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.411+0000 7f2380e4d640 1 -- 172.21.15.57:0/2959735036 >> v1:172.21.15.57:6789/0 conn(0x7f237c0a9d70 legacy=0x7f237c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:50.393 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.411+0000 7f2380e4d640 10 monclient: _finish_hunting 0 2024-03-29T10:48:50.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.411+0000 7f2380e4d640 1 monclient: found mon.noname-b 2024-03-29T10:48:50.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.411+0000 7f2380e4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:50.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.411+0000 7f2380e4d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:48:50.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.411+0000 7f2380e4d640 1 -- 172.21.15.57:0/2959735036 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c146340 con 0x7f237c13f650 2024-03-29T10:48:50.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.411+0000 7f2380e4d640 10 monclient: _finish_auth 0 2024-03-29T10:48:50.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.411+0000 7f2380e4d640 10 monclient: _check_auth_tickets 2024-03-29T10:48:50.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.411+0000 7f2380e4d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:50.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.411+0000 7f2380e4d640 1 -- 172.21.15.57:0/2959735036 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f236c0015f0 con 0x7f237c13f650 2024-03-29T10:48:50.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.411+0000 7f2380e4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:50.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.411+0000 7f2380e4d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:50.394 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.411+0000 7f2380e4d640 10 monclient: dump: 2024-03-29T10:48:50.394 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:50.394 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:50.394 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:50.394 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:50.395 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:50.395 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:50.395 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:50.395 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:50.395 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:50.395 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:50.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.412+0000 7f2380e4d640 1 -- 172.21.15.57:0/2959735036 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f236c0017d0 con 0x7f237c13f650 2024-03-29T10:48:50.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.412+0000 7f2380e4d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:50.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.412+0000 7f2380e4d640 1 -- 172.21.15.57:0/2959735036 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f236c0015f0 con 0x7f237c13f650 2024-03-29T10:48:50.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.412+0000 7f2380e4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:50.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.412+0000 7f2380e4d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:50.396 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.412+0000 7f2380e4d640 10 monclient: dump: 2024-03-29T10:48:50.396 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:50.396 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:50.396 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:50.396 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:50.396 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:50.396 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:50.396 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:50.397 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:50.397 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:50.397 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:50.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.412+0000 7f23840b3640 5 monclient: authenticate success, global_id 4592 2024-03-29T10:48:50.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.412+0000 7f23840b3640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:50.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.412+0000 7f23840b3640 4 set_mon_vals no callback set 2024-03-29T10:48:50.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.413+0000 7f23840b3640 10 monclient: shutdown 2024-03-29T10:48:50.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.413+0000 7f23840b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:50.397 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.413+0000 7f23840b3640 1 -- 172.21.15.57:0/2959735036 >> v1:172.21.15.57:6790/0 conn(0x7f237c13f650 legacy=0x7f237c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:50.398 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.416+0000 7f23840b3640 1 -- 172.21.15.57:0/2959735036 shutdown_connections 2024-03-29T10:48:50.398 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.416+0000 7f23840b3640 1 -- 172.21.15.57:0/2959735036 >> 172.21.15.57:0/2959735036 conn(0x7f237c066a00 msgr2=0x7f237c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:50.398 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.417+0000 7f23840b3640 1 -- 172.21.15.57:0/2959735036 shutdown_connections 2024-03-29T10:48:50.399 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.417+0000 7f23840b3640 1 -- 172.21.15.57:0/2959735036 wait complete. 2024-03-29T10:48:50.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.418+0000 7f23840b3640 10 monclient: build_initial_monmap 2024-03-29T10:48:50.400 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.418+0000 7f23840b3640 10 monclient: monmap: 2024-03-29T10:48:50.400 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:48:50.400 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:50.400 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:48:50.400 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:48:50.400 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:50.400 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:50.400 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:48:50.400 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:48:50.401 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:48:50.401 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:50.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.419+0000 7f23840b3640 1 librados: starting msgr at 2024-03-29T10:48:50.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.419+0000 7f23840b3640 1 librados: starting objecter 2024-03-29T10:48:50.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.419+0000 7f23840b3640 1 Processor -- start 2024-03-29T10:48:50.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.419+0000 7f23840b3640 1 -- start start 2024-03-29T10:48:50.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.419+0000 7f23840b3640 1 librados: setting wanted keys 2024-03-29T10:48:50.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.419+0000 7f23840b3640 1 librados: calling monclient init 2024-03-29T10:48:50.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.419+0000 7f23840b3640 10 monclient: init 2024-03-29T10:48:50.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.420+0000 7f23840b3640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:50.402 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.420+0000 7f23840b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:48:50.405 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.421+0000 7f23840b3640 10 monclient(hunting): picked mon.noname-a con 0x7f237c0a9d70 addr v1:172.21.15.57:6789/0 2024-03-29T10:48:50.405 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.423+0000 7f23840b3640 10 monclient(hunting): picked mon.noname-b con 0x7f237c13f650 addr v1:172.21.15.179:6789/0 2024-03-29T10:48:50.405 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.423+0000 7f23840b3640 10 monclient(hunting): picked mon.noname-c con 0x7f237c1457f0 addr v1:172.21.15.57:6790/0 2024-03-29T10:48:50.405 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.423+0000 7f2382650640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f237c1457f0 0x7f237c1e2500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:42596/0 (socket says 172.21.15.57:42596) 2024-03-29T10:48:50.405 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.423+0000 7f2382650640 1 -- 172.21.15.57:0/2154839861 learned_addr learned my addr 172.21.15.57:0/2154839861 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:48:50.406 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.424+0000 7f23727fc640 10 client.?.objecter ms_handle_connect 0x7f237c1457f0 2024-03-29T10:48:50.406 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.424+0000 7f23840b3640 1 -- 172.21.15.57:0/2154839861 --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c138fb0 con 0x7f237c0a9d70 2024-03-29T10:48:50.410 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.424+0000 7f23727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:50.411 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.424+0000 7f23840b3640 1 -- 172.21.15.57:0/2154839861 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c13aeb0 con 0x7f237c13f650 2024-03-29T10:48:50.411 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.424+0000 7f23840b3640 1 -- 172.21.15.57:0/2154839861 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c13b190 con 0x7f237c1457f0 2024-03-29T10:48:50.411 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.424+0000 7f23840b3640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:50.411 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.424+0000 7f23840b3640 10 monclient(hunting): authenticate will time out at 1482.600350s 2024-03-29T10:48:50.411 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.424+0000 7f23727fc640 10 client.?.objecter ms_handle_connect 0x7f237c13f650 2024-03-29T10:48:50.411 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.424+0000 7f23727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:50.411 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.424+0000 7f23727fc640 1 -- 172.21.15.57:0/2154839861 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349468855 0 0) 0x7f237c13b190 con 0x7f237c1457f0 2024-03-29T10:48:50.411 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.424+0000 7f23727fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:50.411 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.424+0000 7f23727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:50.411 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.424+0000 7f23727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:50.411 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.424+0000 7f23727fc640 10 monclient(hunting): my global_id is 4595 2024-03-29T10:48:50.411 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.424+0000 7f23727fc640 1 -- 172.21.15.57:0/2154839861 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2360004eb0 con 0x7f237c1457f0 2024-03-29T10:48:50.411 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.425+0000 7f23727fc640 1 -- 172.21.15.57:0/2154839861 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318699567 0 0) 0x7f237c13aeb0 con 0x7f237c13f650 2024-03-29T10:48:50.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.425+0000 7f23727fc640 10 monclient(hunting): _negotiate 2024-03-29T10:48:50.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.425+0000 7f23727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:50.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.425+0000 7f23727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:50.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.425+0000 7f23727fc640 10 monclient(hunting): my global_id is 4540 2024-03-29T10:48:50.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.425+0000 7f23727fc640 1 -- 172.21.15.57:0/2154839861 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f237c13b190 con 0x7f237c13f650 2024-03-29T10:48:50.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.426+0000 7f23727fc640 1 -- 172.21.15.57:0/2154839861 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2236747820 0 0) 0x7f2360004eb0 con 0x7f237c1457f0 2024-03-29T10:48:50.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.426+0000 7f23727fc640 1 -- 172.21.15.57:0/2154839861 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f237c13aeb0 con 0x7f237c1457f0 2024-03-29T10:48:50.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.426+0000 7f23727fc640 1 -- 172.21.15.57:0/2154839861 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f2378001950 con 0x7f237c1457f0 2024-03-29T10:48:50.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.426+0000 7f23727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:50.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.426+0000 7f23727fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:50.412 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.426+0000 7f23727fc640 10 monclient(hunting): dump: 2024-03-29T10:48:50.412 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:50.412 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:50.412 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:50.413 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:50.413 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:50.413 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:50.413 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:50.413 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:50.413 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:50.413 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:50.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.426+0000 7f23727fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:50.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.426+0000 7f23727fc640 1 -- 172.21.15.57:0/2154839861 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 387658178 0 0) 0x7f237c13aeb0 con 0x7f237c1457f0 2024-03-29T10:48:50.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.426+0000 7f23727fc640 1 -- 172.21.15.57:0/2154839861 >> v1:172.21.15.179:6789/0 conn(0x7f237c13f650 legacy=0x7f237c1dedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:50.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.426+0000 7f23727fc640 1 -- 172.21.15.57:0/2154839861 >> v1:172.21.15.57:6789/0 conn(0x7f237c0a9d70 legacy=0x7f237c136c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:50.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.426+0000 7f23727fc640 10 monclient: _finish_hunting 0 2024-03-29T10:48:50.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.427+0000 7f23727fc640 1 monclient: found mon.c 2024-03-29T10:48:50.413 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.427+0000 7f23727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:50.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.427+0000 7f23727fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:50.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.427+0000 7f23727fc640 1 -- 172.21.15.57:0/2154839861 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c13b470 con 0x7f237c1457f0 2024-03-29T10:48:50.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.427+0000 7f23727fc640 10 monclient: _finish_auth 0 2024-03-29T10:48:50.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.427+0000 7f23727fc640 10 monclient: _check_auth_tickets 2024-03-29T10:48:50.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.427+0000 7f23727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:50.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.427+0000 7f23840b3640 5 monclient: authenticate success, global_id 4595 2024-03-29T10:48:50.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.427+0000 7f23840b3640 10 monclient: _renew_subs 2024-03-29T10:48:50.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.427+0000 7f23840b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:50.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.427+0000 7f23840b3640 1 -- 172.21.15.57:0/2154839861 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f237c0ab600 con 0x7f237c1457f0 2024-03-29T10:48:50.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.427+0000 7f23840b3640 10 client.4595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:50.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.427+0000 7f23840b3640 10 monclient: _renew_subs 2024-03-29T10:48:50.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.427+0000 7f23840b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:50.414 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.427+0000 7f23840b3640 1 -- 172.21.15.57:0/2154839861 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f237c13ba70 con 0x7f237c1457f0 2024-03-29T10:48:50.415 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.427+0000 7f23840b3640 1 librados: init done 2024-03-29T10:48:50.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.429+0000 7f23727fc640 1 -- 172.21.15.57:0/2154839861 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2378003610 con 0x7f237c1457f0 2024-03-29T10:48:50.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.429+0000 7f23727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:50.416 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.429+0000 7f23727fc640 1 -- 172.21.15.57:0/2154839861 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f2378001ab0 con 0x7f237c1457f0 2024-03-29T10:48:50.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.429+0000 7f23727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:50.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.429+0000 7f23727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:50.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.429+0000 7f23727fc640 10 monclient: dump: 2024-03-29T10:48:50.417 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:50.417 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:50.417 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:50.417 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:50.417 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:50.417 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:50.417 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:50.417 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:50.417 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:50.417 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:50.417 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.429+0000 7f2367fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:50.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.430+0000 7f2367fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:50.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.430+0000 7f2367fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:50.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.430+0000 7f2373fff640 4 set_mon_vals no callback set 2024-03-29T10:48:50.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.430+0000 7f2367fff640 1 -- 172.21.15.57:0/2154839861 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f235c005080 con 0x7f237c1457f0 2024-03-29T10:48:50.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.430+0000 7f23727fc640 1 -- 172.21.15.57:0/2154839861 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f2378003ae0 con 0x7f237c1457f0 2024-03-29T10:48:50.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.431+0000 7f23727fc640 1 -- 172.21.15.57:0/2154839861 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f2378084a40 con 0x7f237c1457f0 2024-03-29T10:48:50.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.431+0000 7f2381e4f640 10 monclient: get_auth_request con 0x7f236006dce0 auth_method 0 2024-03-29T10:48:50.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.431+0000 7f23727fc640 10 client.4595.objecter ms_dispatch 0x7f237c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:50.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.432+0000 7f23727fc640 3 client.4595.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:50.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.432+0000 7f23727fc640 3 client.4595.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:50.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.432+0000 7f23727fc640 20 client.4595.objecter dump_active .. 0 homeless 2024-03-29T10:48:50.418 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.432+0000 7f23727fc640 10 client.4595.objecter ms_handle_connect 0x7f236006dce0 2024-03-29T10:48:50.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.438+0000 7f23727fc640 1 -- 172.21.15.57:0/2154839861 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2378050320 con 0x7f237c1457f0 2024-03-29T10:48:50.420 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.439+0000 7f23727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:50.421 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.439+0000 7f23727fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:50.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.777+0000 7f2367fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:50.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.777+0000 7f2367fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:50.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.777+0000 7f2367fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:48:50.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.778+0000 7f2367fff640 1 -- 172.21.15.57:0/2154839861 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f235c005330 con 0x7f237c1457f0 2024-03-29T10:48:50.762 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.780+0000 7f23727fc640 1 -- 172.21.15.57:0/2154839861 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f2378052250 con 0x7f237c1457f0 2024-03-29T10:48:50.762 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.780+0000 7f23727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:50.762 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.780+0000 7f23727fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:50.762 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:48:50.762 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:50.765 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.783+0000 7f2367fff640 10 librados: watch_flush enter 2024-03-29T10:48:50.766 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.784+0000 7f2367fff640 10 librados: watch_flush exit 2024-03-29T10:48:50.766 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.784+0000 7f2367fff640 20 client.4595.objecter shutdown clearing up homeless session... 2024-03-29T10:48:50.766 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.784+0000 7f2367fff640 10 client.4595.objecter successfully canceled tick 2024-03-29T10:48:50.766 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.784+0000 7f2367fff640 1 -- 172.21.15.57:0/2154839861 >> v1:172.21.15.179:6816/2698295936 conn(0x7f236006dce0 legacy=0x7f23600701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:50.766 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.784+0000 7f2367fff640 10 monclient: shutdown 2024-03-29T10:48:50.766 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.784+0000 7f2367fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:50.766 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.784+0000 7f2367fff640 1 -- 172.21.15.57:0/2154839861 >> v1:172.21.15.57:6790/0 conn(0x7f237c1457f0 legacy=0x7f237c1e2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:50.766 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.784+0000 7f2367fff640 1 -- 172.21.15.57:0/2154839861 shutdown_connections 2024-03-29T10:48:50.766 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.784+0000 7f2367fff640 1 -- 172.21.15.57:0/2154839861 >> 172.21.15.57:0/2154839861 conn(0x7f237c066a00 msgr2=0x7f237c13c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:50.767 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.785+0000 7f2367fff640 1 -- 172.21.15.57:0/2154839861 shutdown_connections 2024-03-29T10:48:50.767 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.785+0000 7f2367fff640 1 -- 172.21.15.57:0/2154839861 wait complete. 2024-03-29T10:48:50.768 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:50.786+0000 7f2367fff640 1 librados: shutdown 2024-03-29T10:48:50.784 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:48:57.592 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:48:57.737 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.754+0000 7fc176efb640 10 monclient: get_monmap_and_config 2024-03-29T10:48:57.737 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.754+0000 7fc176efb640 10 monclient: build_initial_monmap 2024-03-29T10:48:57.737 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.754+0000 7fc176efb640 10 monclient: monmap: 2024-03-29T10:48:57.737 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:48:57.737 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:57.737 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:48:57.755631+0000 2024-03-29T10:48:57.737 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:48:57.755631+0000 2024-03-29T10:48:57.737 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:57.737 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:57.737 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:48:57.737 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:48:57.738 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:48:57.738 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:57.742 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.756+0000 7fc176efb640 1 Processor -- start 2024-03-29T10:48:57.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.756+0000 7fc176efb640 1 -- start start 2024-03-29T10:48:57.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.756+0000 7fc176efb640 10 monclient: init 2024-03-29T10:48:57.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.756+0000 7fc176efb640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:57.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.756+0000 7fc176efb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:48:57.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.757+0000 7fc176efb640 10 monclient(hunting): picked mon.noname-a con 0x7fc1701422b0 addr v1:172.21.15.57:6789/0 2024-03-29T10:48:57.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.757+0000 7fc176efb640 10 monclient(hunting): picked mon.noname-c con 0x7fc170142690 addr v1:172.21.15.179:6789/0 2024-03-29T10:48:57.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.757+0000 7fc176efb640 10 monclient(hunting): picked mon.noname-b con 0x7fc1700b17d0 addr v1:172.21.15.57:6790/0 2024-03-29T10:48:57.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.757+0000 7fc176efb640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc170142c80 con 0x7fc1701422b0 2024-03-29T10:48:57.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.757+0000 7fc176efb640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1700aae40 con 0x7fc170142690 2024-03-29T10:48:57.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.757+0000 7fc176efb640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc17013fbd0 con 0x7fc1700b17d0 2024-03-29T10:48:57.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.757+0000 7fc176efb640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:57.743 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.757+0000 7fc176efb640 10 monclient(hunting): authenticate will time out at 1489.933102s 2024-03-29T10:48:57.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.757+0000 7fc175498640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7fc1700b17d0 0x7fc1700b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:45738/0 (socket says 172.21.15.57:45738) 2024-03-29T10:48:57.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.757+0000 7fc175498640 1 -- 172.21.15.57:0/1613593614 learned_addr learned my addr 172.21.15.57:0/1613593614 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:48:57.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 1 -- 172.21.15.57:0/1613593614 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615791007 0 0) 0x7fc17013fbd0 con 0x7fc1700b17d0 2024-03-29T10:48:57.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:57.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:57.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:57.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 10 monclient(hunting): my global_id is 4598 2024-03-29T10:48:57.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 1 -- 172.21.15.57:0/1613593614 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc160003520 con 0x7fc1700b17d0 2024-03-29T10:48:57.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 1 -- 172.21.15.57:0/1613593614 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599421620 0 0) 0x7fc1700aae40 con 0x7fc170142690 2024-03-29T10:48:57.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:57.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:57.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:57.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 10 monclient(hunting): my global_id is 4543 2024-03-29T10:48:57.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 1 -- 172.21.15.57:0/1613593614 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc17013fbd0 con 0x7fc170142690 2024-03-29T10:48:57.744 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 1 -- 172.21.15.57:0/1613593614 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 905669985 0 0) 0x7fc170142c80 con 0x7fc1701422b0 2024-03-29T10:48:57.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:48:57.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:57.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:57.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 10 monclient(hunting): my global_id is 4539 2024-03-29T10:48:57.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 1 -- 172.21.15.57:0/1613593614 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1700aae40 con 0x7fc1701422b0 2024-03-29T10:48:57.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 1 -- 172.21.15.57:0/1613593614 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3965667667 0 0) 0x7fc160003520 con 0x7fc1700b17d0 2024-03-29T10:48:57.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 1 -- 172.21.15.57:0/1613593614 >> v1:172.21.15.179:6789/0 conn(0x7fc170142690 legacy=0x7fc1700b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:57.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 1 -- 172.21.15.57:0/1613593614 >> v1:172.21.15.57:6789/0 conn(0x7fc1701422b0 legacy=0x7fc17013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:57.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:48:57.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 1 monclient: found mon.noname-b 2024-03-29T10:48:57.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:57.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:48:57.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 1 -- 172.21.15.57:0/1613593614 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc170151940 con 0x7fc1700b17d0 2024-03-29T10:48:57.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:48:57.745 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:48:57.746 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:57.746 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 1 -- 172.21.15.57:0/1613593614 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fc168001a10 con 0x7fc1700b17d0 2024-03-29T10:48:57.746 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:57.746 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:48:57.746 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 10 monclient: dump: 2024-03-29T10:48:57.746 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:57.746 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:57.746 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:57.746 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:57.746 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:57.746 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:57.746 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:57.746 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:57.746 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:57.747 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:57.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc176efb640 5 monclient: authenticate success, global_id 4598 2024-03-29T10:48:57.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc176efb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:48:57.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 1 -- 172.21.15.57:0/1613593614 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc168001b70 con 0x7fc1700b17d0 2024-03-29T10:48:57.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:57.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.758+0000 7fc16f7fe640 1 -- 172.21.15.57:0/1613593614 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fc168001f00 con 0x7fc1700b17d0 2024-03-29T10:48:57.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.759+0000 7fc16f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:57.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.759+0000 7fc16f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:48:57.747 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.759+0000 7fc16f7fe640 10 monclient: dump: 2024-03-29T10:48:57.747 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:57.747 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:57.747 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:57.747 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:57.747 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:57.747 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:57.748 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:57.748 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:57.748 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:57.748 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:57.748 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.759+0000 7fc176efb640 10 monclient: get_monmap_and_config success 2024-03-29T10:48:57.748 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.759+0000 7fc176efb640 4 set_mon_vals no callback set 2024-03-29T10:48:57.748 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.759+0000 7fc176efb640 10 monclient: shutdown 2024-03-29T10:48:57.748 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.759+0000 7fc176efb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:57.748 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.759+0000 7fc176efb640 1 -- 172.21.15.57:0/1613593614 >> v1:172.21.15.57:6790/0 conn(0x7fc1700b17d0 legacy=0x7fc1700b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:57.748 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.760+0000 7fc176efb640 1 -- 172.21.15.57:0/1613593614 shutdown_connections 2024-03-29T10:48:57.748 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.760+0000 7fc176efb640 1 -- 172.21.15.57:0/1613593614 >> 172.21.15.57:0/1613593614 conn(0x7fc170066a00 msgr2=0x7fc170066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:57.748 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.760+0000 7fc176efb640 1 -- 172.21.15.57:0/1613593614 shutdown_connections 2024-03-29T10:48:57.748 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.760+0000 7fc176efb640 1 -- 172.21.15.57:0/1613593614 wait complete. 2024-03-29T10:48:57.748 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.761+0000 7fc176efb640 10 monclient: build_initial_monmap 2024-03-29T10:48:57.748 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.761+0000 7fc176efb640 10 monclient: monmap: 2024-03-29T10:48:57.749 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:48:57.749 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:48:57.749 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:48:57.749 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:48:57.749 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:48:57.749 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:57.749 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:48:57.749 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:48:57.749 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:48:57.749 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:57.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.761+0000 7fc176efb640 1 librados: starting msgr at 2024-03-29T10:48:57.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.761+0000 7fc176efb640 1 librados: starting objecter 2024-03-29T10:48:57.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.762+0000 7fc176efb640 1 Processor -- start 2024-03-29T10:48:57.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.762+0000 7fc176efb640 1 -- start start 2024-03-29T10:48:57.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.762+0000 7fc176efb640 1 librados: setting wanted keys 2024-03-29T10:48:57.756 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.762+0000 7fc176efb640 1 librados: calling monclient init 2024-03-29T10:48:57.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.762+0000 7fc176efb640 10 monclient: init 2024-03-29T10:48:57.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.763+0000 7fc176efb640 10 monclient: _reopen_session rank -1 2024-03-29T10:48:57.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.763+0000 7fc176efb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:48:57.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.763+0000 7fc176efb640 10 monclient(hunting): picked mon.noname-b con 0x7fc1700b17d0 addr v1:172.21.15.179:6789/0 2024-03-29T10:48:57.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.763+0000 7fc176efb640 10 monclient(hunting): picked mon.noname-a con 0x7fc1701422b0 addr v1:172.21.15.57:6789/0 2024-03-29T10:48:57.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.763+0000 7fc176efb640 10 monclient(hunting): picked mon.noname-c con 0x7fc170142690 addr v1:172.21.15.57:6790/0 2024-03-29T10:48:57.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.763+0000 7fc176efb640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc170071040 con 0x7fc1701422b0 2024-03-29T10:48:57.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.763+0000 7fc176efb640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc170071320 con 0x7fc1700b17d0 2024-03-29T10:48:57.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.763+0000 7fc176efb640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1701dbe20 con 0x7fc170142690 2024-03-29T10:48:57.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.763+0000 7fc176efb640 10 monclient(hunting): _renew_subs 2024-03-29T10:48:57.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.763+0000 7fc176efb640 10 monclient(hunting): authenticate will time out at 1489.939493s 2024-03-29T10:48:57.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.763+0000 7fc174c97640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7fc1700b17d0 0x7fc17006ec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:60214/0 (socket says 172.21.15.57:60214) 2024-03-29T10:48:57.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.763+0000 7fc174c97640 1 -- 172.21.15.57:0/1713413171 learned_addr learned my addr 172.21.15.57:0/1713413171 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:48:57.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.763+0000 7fc16d7fa640 10 client.?.objecter ms_handle_connect 0x7fc170142690 2024-03-29T10:48:57.757 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.763+0000 7fc16d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:57.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.763+0000 7fc16d7fa640 10 client.?.objecter ms_handle_connect 0x7fc1700b17d0 2024-03-29T10:48:57.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.763+0000 7fc16d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:57.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 1 -- 172.21.15.57:0/1713413171 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813237169 0 0) 0x7fc170071320 con 0x7fc1700b17d0 2024-03-29T10:48:57.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:48:57.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:57.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:57.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 10 monclient(hunting): my global_id is 4546 2024-03-29T10:48:57.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 1 -- 172.21.15.57:0/1713413171 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc154004eb0 con 0x7fc1700b17d0 2024-03-29T10:48:57.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 1 -- 172.21.15.57:0/1713413171 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695872393 0 0) 0x7fc1701dbe20 con 0x7fc170142690 2024-03-29T10:48:57.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:48:57.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:57.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:57.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 10 monclient(hunting): my global_id is 4601 2024-03-29T10:48:57.758 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 1 -- 172.21.15.57:0/1713413171 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc170071320 con 0x7fc170142690 2024-03-29T10:48:57.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 10 client.?.objecter ms_handle_connect 0x7fc1701422b0 2024-03-29T10:48:57.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:48:57.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 1 -- 172.21.15.57:0/1713413171 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699554695 0 0) 0x7fc170071040 con 0x7fc1701422b0 2024-03-29T10:48:57.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:48:57.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:48:57.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:48:57.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 10 monclient(hunting): my global_id is 4542 2024-03-29T10:48:57.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 1 -- 172.21.15.57:0/1713413171 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1701dbe20 con 0x7fc1701422b0 2024-03-29T10:48:57.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 1 -- 172.21.15.57:0/1713413171 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4266469106 0 0) 0x7fc170071320 con 0x7fc170142690 2024-03-29T10:48:57.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 1 -- 172.21.15.57:0/1713413171 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc170071040 con 0x7fc170142690 2024-03-29T10:48:57.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 1 -- 172.21.15.57:0/1713413171 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fc168001ce0 con 0x7fc170142690 2024-03-29T10:48:57.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:57.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:48:57.759 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 10 monclient(hunting): dump: 2024-03-29T10:48:57.759 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:57.760 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:57.760 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:57.760 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:57.760 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:57.760 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:57.760 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:57.760 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:57.760 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:57.760 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:57.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.764+0000 7fc16d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:48:57.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.765+0000 7fc16d7fa640 1 -- 172.21.15.57:0/1713413171 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1446260636 0 0) 0x7fc1701dbe20 con 0x7fc1701422b0 2024-03-29T10:48:57.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.765+0000 7fc16d7fa640 1 -- 172.21.15.57:0/1713413171 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc170071320 con 0x7fc1701422b0 2024-03-29T10:48:57.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.765+0000 7fc16d7fa640 1 -- 172.21.15.57:0/1713413171 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fc15c001850 con 0x7fc1701422b0 2024-03-29T10:48:57.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.765+0000 7fc16d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:57.760 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.765+0000 7fc16d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:57.761 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.765+0000 7fc16d7fa640 10 monclient(hunting): dump: 2024-03-29T10:48:57.761 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:57.761 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:57.761 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:57.761 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:57.761 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:57.761 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:57.761 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:57.761 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:57.761 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:57.761 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:57.761 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.765+0000 7fc16d7fa640 1 -- 172.21.15.57:0/1713413171 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1162520744 0 0) 0x7fc170071320 con 0x7fc1701422b0 2024-03-29T10:48:57.761 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.765+0000 7fc16d7fa640 1 -- 172.21.15.57:0/1713413171 >> v1:172.21.15.57:6790/0 conn(0x7fc170142690 legacy=0x7fc1701e7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:57.761 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.765+0000 7fc16d7fa640 1 -- 172.21.15.57:0/1713413171 >> v1:172.21.15.179:6789/0 conn(0x7fc1700b17d0 legacy=0x7fc17006ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:57.761 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.765+0000 7fc16d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:48:57.762 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.765+0000 7fc16d7fa640 1 monclient: found mon.a 2024-03-29T10:48:57.762 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.765+0000 7fc16d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:48:57.762 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.765+0000 7fc16d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:48:57.762 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.765+0000 7fc16d7fa640 1 -- 172.21.15.57:0/1713413171 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17013c9b0 con 0x7fc1701422b0 2024-03-29T10:48:57.762 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.765+0000 7fc16d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:48:57.762 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.765+0000 7fc16d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:48:57.762 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.766+0000 7fc16d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:48:57.762 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.766+0000 7fc176efb640 5 monclient: authenticate success, global_id 4542 2024-03-29T10:48:57.762 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.766+0000 7fc176efb640 10 monclient: _renew_subs 2024-03-29T10:48:57.762 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.766+0000 7fc176efb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:48:57.762 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.766+0000 7fc176efb640 1 -- 172.21.15.57:0/1713413171 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1700b0bf0 con 0x7fc1701422b0 2024-03-29T10:48:57.762 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.766+0000 7fc176efb640 10 client.4542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:48:57.762 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.766+0000 7fc176efb640 10 monclient: _renew_subs 2024-03-29T10:48:57.762 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.766+0000 7fc176efb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:48:57.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.766+0000 7fc176efb640 1 -- 172.21.15.57:0/1713413171 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc170071c90 con 0x7fc1701422b0 2024-03-29T10:48:57.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.766+0000 7fc176efb640 1 librados: init done 2024-03-29T10:48:57.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.766+0000 7fc16d7fa640 1 -- 172.21.15.57:0/1713413171 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc15c001d60 con 0x7fc1701422b0 2024-03-29T10:48:57.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.766+0000 7fc16d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:48:57.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.767+0000 7fc16effd640 4 set_mon_vals no callback set 2024-03-29T10:48:57.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.767+0000 7fc176efb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:57.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.767+0000 7fc16d7fa640 1 -- 172.21.15.57:0/1713413171 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fc15c003790 con 0x7fc1701422b0 2024-03-29T10:48:57.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.767+0000 7fc16d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:48:57.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.767+0000 7fc16d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:48:57.763 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.767+0000 7fc16d7fa640 10 monclient: dump: 2024-03-29T10:48:57.763 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:48:57.763 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:48:57.763 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:57.763 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:48:57.764 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:48:57.764 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:48:57.764 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:48:57.764 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:48:57.764 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:48:57.764 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:48:57.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.767+0000 7fc176efb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:57.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.768+0000 7fc176efb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:48:57.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.768+0000 7fc176efb640 1 -- 172.21.15.57:0/1713413171 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc150005080 con 0x7fc1701422b0 2024-03-29T10:48:57.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.770+0000 7fc16d7fa640 1 -- 172.21.15.57:0/1713413171 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fc15c019760 con 0x7fc1701422b0 2024-03-29T10:48:57.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.771+0000 7fc16d7fa640 1 -- 172.21.15.57:0/1713413171 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7fc15c083610 con 0x7fc1701422b0 2024-03-29T10:48:57.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.771+0000 7fc174c97640 10 monclient: get_auth_request con 0x7fc15406ddd0 auth_method 0 2024-03-29T10:48:57.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.771+0000 7fc16d7fa640 10 client.4542.objecter ms_dispatch 0x7fc170000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:48:57.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.771+0000 7fc16d7fa640 3 client.4542.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:48:57.764 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.771+0000 7fc16d7fa640 3 client.4542.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:48:57.765 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.771+0000 7fc16d7fa640 20 client.4542.objecter dump_active .. 0 homeless 2024-03-29T10:48:57.765 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.771+0000 7fc16d7fa640 10 client.4542.objecter ms_handle_connect 0x7fc15406ddd0 2024-03-29T10:48:57.765 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.780+0000 7fc16d7fa640 1 -- 172.21.15.57:0/1713413171 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc15c04eef0 con 0x7fc1701422b0 2024-03-29T10:48:57.765 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.780+0000 7fc16d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:48:57.765 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:57.780+0000 7fc16d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:48:58.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:58.116+0000 7fc176efb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:58.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:58.116+0000 7fc176efb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:58.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:58.116+0000 7fc176efb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:48:58.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:58.116+0000 7fc176efb640 1 -- 172.21.15.57:0/1713413171 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc150005330 con 0x7fc1701422b0 2024-03-29T10:48:58.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:58.117+0000 7fc16d7fa640 1 -- 172.21.15.57:0/1713413171 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7fc15c050e20 con 0x7fc1701422b0 2024-03-29T10:48:58.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:58.117+0000 7fc16d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:48:58.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:58.117+0000 7fc16d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:48:58.099 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:48:58.099 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:48:58.103 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:58.121+0000 7fc176efb640 10 librados: watch_flush enter 2024-03-29T10:48:58.103 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:58.121+0000 7fc176efb640 10 librados: watch_flush exit 2024-03-29T10:48:58.103 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:58.121+0000 7fc176efb640 20 client.4542.objecter shutdown clearing up homeless session... 2024-03-29T10:48:58.103 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:58.121+0000 7fc176efb640 10 client.4542.objecter successfully canceled tick 2024-03-29T10:48:58.103 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:58.121+0000 7fc176efb640 1 -- 172.21.15.57:0/1713413171 >> v1:172.21.15.179:6816/2698295936 conn(0x7fc15406ddd0 legacy=0x7fc154070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:58.103 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:58.121+0000 7fc176efb640 10 monclient: shutdown 2024-03-29T10:48:58.103 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:58.121+0000 7fc176efb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:48:58.103 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:58.121+0000 7fc176efb640 1 -- 172.21.15.57:0/1713413171 >> v1:172.21.15.57:6789/0 conn(0x7fc1701422b0 legacy=0x7fc17006f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:48:58.103 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:58.121+0000 7fc176efb640 1 -- 172.21.15.57:0/1713413171 shutdown_connections 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:58.121+0000 7fc176efb640 1 -- 172.21.15.57:0/1713413171 >> 172.21.15.57:0/1713413171 conn(0x7fc170066a00 msgr2=0x7fc17013ea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:58.121+0000 7fc176efb640 1 -- 172.21.15.57:0/1713413171 shutdown_connections 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:58.121+0000 7fc176efb640 1 -- 172.21.15.57:0/1713413171 wait complete. 2024-03-29T10:48:58.104 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:48:58.121+0000 7fc176efb640 1 librados: shutdown 2024-03-29T10:48:58.118 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:04.926 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:05.091 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.108+0000 7f474d596640 10 monclient: get_monmap_and_config 2024-03-29T10:49:05.091 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.108+0000 7f474d596640 10 monclient: build_initial_monmap 2024-03-29T10:49:05.092 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.108+0000 7f474d596640 10 monclient: monmap: 2024-03-29T10:49:05.092 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:49:05.092 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:05.092 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:49:05.110473+0000 2024-03-29T10:49:05.092 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:49:05.110473+0000 2024-03-29T10:49:05.092 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:05.092 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:05.092 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:49:05.092 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:49:05.092 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:49:05.092 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:05.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.109+0000 7f474d596640 1 Processor -- start 2024-03-29T10:49:05.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.110+0000 7f474d596640 1 -- start start 2024-03-29T10:49:05.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.110+0000 7f474d596640 10 monclient: init 2024-03-29T10:49:05.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.110+0000 7f474d596640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:05.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.110+0000 7f474d596640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:49:05.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.110+0000 7f474d596640 10 monclient(hunting): picked mon.noname-c con 0x7f47481457f0 addr v1:172.21.15.179:6789/0 2024-03-29T10:49:05.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.111+0000 7f474d596640 10 monclient(hunting): picked mon.noname-b con 0x7f474813f650 addr v1:172.21.15.57:6790/0 2024-03-29T10:49:05.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.111+0000 7f474d596640 10 monclient(hunting): picked mon.noname-a con 0x7f47480a9d70 addr v1:172.21.15.57:6789/0 2024-03-29T10:49:05.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.111+0000 7f474d596640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47480aaef0 con 0x7f47480a9d70 2024-03-29T10:49:05.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.111+0000 7f474d596640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4748145bd0 con 0x7f47481457f0 2024-03-29T10:49:05.095 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.111+0000 7f474d596640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4748145eb0 con 0x7f474813f650 2024-03-29T10:49:05.095 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.111+0000 7f474d596640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:05.095 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.111+0000 7f474d596640 10 monclient(hunting): authenticate will time out at 1497.288113s 2024-03-29T10:49:05.095 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.111+0000 7f47477fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f47480a9d70 0x7f474814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:36002/0 (socket says 172.21.15.57:36002) 2024-03-29T10:49:05.095 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.111+0000 7f47477fe640 1 -- 172.21.15.57:0/1349199195 learned_addr learned my addr 172.21.15.57:0/1349199195 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:49:05.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 1 -- 172.21.15.57:0/1349199195 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681909089 0 0) 0x7f4748145bd0 con 0x7f47481457f0 2024-03-29T10:49:05.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:49:05.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:05.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:05.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 10 monclient(hunting): my global_id is 4549 2024-03-29T10:49:05.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 1 -- 172.21.15.57:0/1349199195 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4730003520 con 0x7f47481457f0 2024-03-29T10:49:05.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 1 -- 172.21.15.57:0/1349199195 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1611971951 0 0) 0x7f4748145eb0 con 0x7f474813f650 2024-03-29T10:49:05.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:49:05.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:05.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:05.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 10 monclient(hunting): my global_id is 4604 2024-03-29T10:49:05.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 1 -- 172.21.15.57:0/1349199195 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4748145bd0 con 0x7f474813f650 2024-03-29T10:49:05.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 1 -- 172.21.15.57:0/1349199195 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 459168394 0 0) 0x7f4730003520 con 0x7f47481457f0 2024-03-29T10:49:05.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 1 -- 172.21.15.57:0/1349199195 >> v1:172.21.15.57:6790/0 conn(0x7f474813f650 legacy=0x7f47480b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:05.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 1 -- 172.21.15.57:0/1349199195 >> v1:172.21.15.57:6789/0 conn(0x7f47480a9d70 legacy=0x7f474814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:05.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:49:05.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 1 monclient: found mon.noname-c 2024-03-29T10:49:05.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:05.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:49:05.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 1 -- 172.21.15.57:0/1349199195 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4748146340 con 0x7f47481457f0 2024-03-29T10:49:05.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 10 monclient: _finish_auth 0 2024-03-29T10:49:05.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:49:05.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:05.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 1 -- 172.21.15.57:0/1349199195 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f4738001690 con 0x7f47481457f0 2024-03-29T10:49:05.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:05.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:05.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.112+0000 7f4745ffb640 10 monclient: dump: 2024-03-29T10:49:05.100 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:05.100 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:05.100 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:05.100 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:05.100 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:05.100 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:05.100 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:05.100 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:05.100 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:05.100 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:05.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.113+0000 7f474d596640 5 monclient: authenticate success, global_id 4549 2024-03-29T10:49:05.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.113+0000 7f474d596640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:05.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.113+0000 7f4745ffb640 1 -- 172.21.15.57:0/1349199195 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47380017f0 con 0x7f47481457f0 2024-03-29T10:49:05.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.113+0000 7f4745ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:05.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.113+0000 7f474d596640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:05.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.113+0000 7f474d596640 4 set_mon_vals no callback set 2024-03-29T10:49:05.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.113+0000 7f4745ffb640 1 -- 172.21.15.57:0/1349199195 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f4738001c00 con 0x7f47481457f0 2024-03-29T10:49:05.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.113+0000 7f4745ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:05.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.113+0000 7f4745ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:05.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.113+0000 7f4745ffb640 10 monclient: dump: 2024-03-29T10:49:05.101 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:05.101 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:05.101 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:05.101 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:05.101 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:05.101 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:05.101 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:05.101 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:05.101 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:05.101 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:05.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.114+0000 7f474d596640 10 monclient: shutdown 2024-03-29T10:49:05.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.114+0000 7f474d596640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:05.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.114+0000 7f474d596640 1 -- 172.21.15.57:0/1349199195 >> v1:172.21.15.179:6789/0 conn(0x7f47481457f0 legacy=0x7f474813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:05.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.114+0000 7f474d596640 1 -- 172.21.15.57:0/1349199195 shutdown_connections 2024-03-29T10:49:05.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.114+0000 7f474d596640 1 -- 172.21.15.57:0/1349199195 >> 172.21.15.57:0/1349199195 conn(0x7f4748066a00 msgr2=0x7f4748066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:05.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.114+0000 7f474d596640 1 -- 172.21.15.57:0/1349199195 shutdown_connections 2024-03-29T10:49:05.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.114+0000 7f474d596640 1 -- 172.21.15.57:0/1349199195 wait complete. 2024-03-29T10:49:05.106 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.115+0000 7f474d596640 10 monclient: build_initial_monmap 2024-03-29T10:49:05.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.115+0000 7f474d596640 10 monclient: monmap: 2024-03-29T10:49:05.107 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:49:05.107 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:05.107 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:49:05.107 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:49:05.107 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:05.107 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:05.107 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:49:05.107 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:49:05.107 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:49:05.107 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:05.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.116+0000 7f474d596640 1 librados: starting msgr at 2024-03-29T10:49:05.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.116+0000 7f474d596640 1 librados: starting objecter 2024-03-29T10:49:05.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.116+0000 7f474d596640 1 Processor -- start 2024-03-29T10:49:05.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.116+0000 7f474d596640 1 -- start start 2024-03-29T10:49:05.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.116+0000 7f474d596640 1 librados: setting wanted keys 2024-03-29T10:49:05.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.116+0000 7f474d596640 1 librados: calling monclient init 2024-03-29T10:49:05.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.116+0000 7f474d596640 10 monclient: init 2024-03-29T10:49:05.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.117+0000 7f474d596640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:05.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.117+0000 7f474d596640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:49:05.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.117+0000 7f474d596640 10 monclient(hunting): picked mon.noname-a con 0x7f47480a9d70 addr v1:172.21.15.57:6789/0 2024-03-29T10:49:05.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.117+0000 7f474d596640 10 monclient(hunting): picked mon.noname-c con 0x7f474813f650 addr v1:172.21.15.57:6790/0 2024-03-29T10:49:05.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.117+0000 7f474d596640 10 monclient(hunting): picked mon.noname-b con 0x7f47481457f0 addr v1:172.21.15.179:6789/0 2024-03-29T10:49:05.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.117+0000 7f474d596640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4748071e20 con 0x7f47480a9d70 2024-03-29T10:49:05.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.117+0000 7f474d596640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f474806fc70 con 0x7f47481457f0 2024-03-29T10:49:05.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.117+0000 7f474d596640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f474806eae0 con 0x7f474813f650 2024-03-29T10:49:05.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.117+0000 7f474d596640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:05.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.117+0000 7f474d596640 10 monclient(hunting): authenticate will time out at 1497.294907s 2024-03-29T10:49:05.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.118+0000 7f47467fc640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f474813f650 0x7f4748071710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:40542/0 (socket says 172.21.15.57:40542) 2024-03-29T10:49:05.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.118+0000 7f47467fc640 1 -- 172.21.15.57:0/657356260 learned_addr learned my addr 172.21.15.57:0/657356260 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:49:05.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.118+0000 7f472f7fe640 10 client.?.objecter ms_handle_connect 0x7f474813f650 2024-03-29T10:49:05.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.118+0000 7f472f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:05.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.118+0000 7f472f7fe640 10 client.?.objecter ms_handle_connect 0x7f47480a9d70 2024-03-29T10:49:05.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.118+0000 7f472f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:05.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.118+0000 7f472f7fe640 1 -- 172.21.15.57:0/657356260 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456487794 0 0) 0x7f4748071e20 con 0x7f47480a9d70 2024-03-29T10:49:05.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.119+0000 7f472f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:05.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.119+0000 7f472f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:05.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.119+0000 7f472f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:05.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.119+0000 7f472f7fe640 10 monclient(hunting): my global_id is 4548 2024-03-29T10:49:05.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.119+0000 7f472f7fe640 1 -- 172.21.15.57:0/657356260 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4734004eb0 con 0x7f47480a9d70 2024-03-29T10:49:05.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.119+0000 7f472f7fe640 10 client.?.objecter ms_handle_connect 0x7f47481457f0 2024-03-29T10:49:05.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.119+0000 7f472f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:05.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.119+0000 7f472f7fe640 1 -- 172.21.15.57:0/657356260 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 277249063 0 0) 0x7f474806eae0 con 0x7f474813f650 2024-03-29T10:49:05.109 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.119+0000 7f472f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:05.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.119+0000 7f472f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:05.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.119+0000 7f472f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:05.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.119+0000 7f472f7fe640 10 monclient(hunting): my global_id is 4607 2024-03-29T10:49:05.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.119+0000 7f472f7fe640 1 -- 172.21.15.57:0/657356260 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4748071e20 con 0x7f474813f650 2024-03-29T10:49:05.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.119+0000 7f472f7fe640 1 -- 172.21.15.57:0/657356260 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2240670156 0 0) 0x7f4734004eb0 con 0x7f47480a9d70 2024-03-29T10:49:05.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.119+0000 7f472f7fe640 1 -- 172.21.15.57:0/657356260 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f474806eae0 con 0x7f47480a9d70 2024-03-29T10:49:05.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.119+0000 7f472f7fe640 1 -- 172.21.15.57:0/657356260 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 67242036 0 0) 0x7f474806fc70 con 0x7f47481457f0 2024-03-29T10:49:05.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f472f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:05.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f472f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:05.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f472f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:05.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f472f7fe640 10 monclient(hunting): my global_id is 4552 2024-03-29T10:49:05.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f472f7fe640 1 -- 172.21.15.57:0/657356260 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4734004eb0 con 0x7f47481457f0 2024-03-29T10:49:05.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f472f7fe640 1 -- 172.21.15.57:0/657356260 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f4738001b40 con 0x7f47480a9d70 2024-03-29T10:49:05.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f472f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:05.110 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f472f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:05.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f472f7fe640 10 monclient(hunting): dump: 2024-03-29T10:49:05.111 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:05.111 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:05.111 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:05.111 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:05.111 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:05.111 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:05.111 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:05.111 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:05.111 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:05.111 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:05.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f472f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:05.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f472f7fe640 1 -- 172.21.15.57:0/657356260 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 496682598 0 0) 0x7f474806eae0 con 0x7f47480a9d70 2024-03-29T10:49:05.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f472f7fe640 1 -- 172.21.15.57:0/657356260 >> v1:172.21.15.57:6790/0 conn(0x7f474813f650 legacy=0x7f4748071710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:05.111 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f472f7fe640 1 -- 172.21.15.57:0/657356260 >> v1:172.21.15.179:6789/0 conn(0x7f47481457f0 legacy=0x7f474813fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:05.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f472f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:49:05.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f472f7fe640 1 monclient: found mon.a 2024-03-29T10:49:05.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f472f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:05.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f472f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:49:05.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f472f7fe640 1 -- 172.21.15.57:0/657356260 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474806e680 con 0x7f47480a9d70 2024-03-29T10:49:05.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f472f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:49:05.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f472f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:49:05.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f472f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:05.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f474d596640 5 monclient: authenticate success, global_id 4548 2024-03-29T10:49:05.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f474d596640 10 monclient: _renew_subs 2024-03-29T10:49:05.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f474d596640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:49:05.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f474d596640 1 -- 172.21.15.57:0/657356260 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4748142970 con 0x7f47480a9d70 2024-03-29T10:49:05.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f474d596640 10 client.4548.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:05.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f474d596640 10 monclient: _renew_subs 2024-03-29T10:49:05.112 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f474d596640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:49:05.113 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f474d596640 1 -- 172.21.15.57:0/657356260 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47481405b0 con 0x7f47480a9d70 2024-03-29T10:49:05.113 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.120+0000 7f474d596640 1 librados: init done 2024-03-29T10:49:05.113 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.121+0000 7f472f7fe640 1 -- 172.21.15.57:0/657356260 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47380038a0 con 0x7f47480a9d70 2024-03-29T10:49:05.113 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.121+0000 7f472f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:05.113 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.121+0000 7f472f7fe640 1 -- 172.21.15.57:0/657356260 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f4738001ca0 con 0x7f47480a9d70 2024-03-29T10:49:05.113 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.121+0000 7f47457fa640 4 set_mon_vals no callback set 2024-03-29T10:49:05.113 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.121+0000 7f472f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:05.113 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.121+0000 7f472f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:05.113 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.121+0000 7f472f7fe640 10 monclient: dump: 2024-03-29T10:49:05.113 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:05.113 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:05.113 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:05.113 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:05.113 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:05.113 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:05.114 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:05.114 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:05.114 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:05.114 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:05.114 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.122+0000 7f474d596640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:05.114 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.122+0000 7f474d596640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:05.114 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.122+0000 7f474d596640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:49:05.114 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.122+0000 7f474d596640 1 -- 172.21.15.57:0/657356260 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4728005080 con 0x7f47480a9d70 2024-03-29T10:49:05.114 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.123+0000 7f472f7fe640 1 -- 172.21.15.57:0/657356260 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f4738019a20 con 0x7f47480a9d70 2024-03-29T10:49:05.115 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.124+0000 7f472f7fe640 1 -- 172.21.15.57:0/657356260 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f47380838c0 con 0x7f47480a9d70 2024-03-29T10:49:05.115 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.124+0000 7f472f7fe640 10 client.4548.objecter ms_dispatch 0x7f4748000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:05.115 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.124+0000 7f472f7fe640 3 client.4548.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:05.115 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.124+0000 7f472f7fe640 3 client.4548.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:05.115 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.124+0000 7f472f7fe640 20 client.4548.objecter dump_active .. 0 homeless 2024-03-29T10:49:05.115 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.125+0000 7f47467fc640 10 monclient: get_auth_request con 0x7f473406ded0 auth_method 0 2024-03-29T10:49:05.115 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.125+0000 7f472f7fe640 10 client.4548.objecter ms_handle_connect 0x7f473406ded0 2024-03-29T10:49:05.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.134+0000 7f472f7fe640 1 -- 172.21.15.57:0/657356260 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f473804f1a0 con 0x7f47480a9d70 2024-03-29T10:49:05.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.134+0000 7f472f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:05.117 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.134+0000 7f472f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:05.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.604+0000 7f474d596640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:05.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.604+0000 7f474d596640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:05.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.605+0000 7f474d596640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:49:05.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.605+0000 7f474d596640 1 -- 172.21.15.57:0/657356260 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4728005330 con 0x7f47480a9d70 2024-03-29T10:49:05.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.606+0000 7f472f7fe640 1 -- 172.21.15.57:0/657356260 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f47380510d0 con 0x7f47480a9d70 2024-03-29T10:49:05.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.606+0000 7f472f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:05.588 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.606+0000 7f472f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:05.589 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:49:05.589 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:05.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.609+0000 7f474d596640 10 librados: watch_flush enter 2024-03-29T10:49:05.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.609+0000 7f474d596640 10 librados: watch_flush exit 2024-03-29T10:49:05.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.609+0000 7f474d596640 20 client.4548.objecter shutdown clearing up homeless session... 2024-03-29T10:49:05.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.609+0000 7f474d596640 10 client.4548.objecter successfully canceled tick 2024-03-29T10:49:05.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.610+0000 7f474d596640 1 -- 172.21.15.57:0/657356260 >> v1:172.21.15.179:6816/2698295936 conn(0x7f473406ded0 legacy=0x7f4734070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:05.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.610+0000 7f474d596640 10 monclient: shutdown 2024-03-29T10:49:05.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.610+0000 7f474d596640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:05.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.610+0000 7f474d596640 1 -- 172.21.15.57:0/657356260 >> v1:172.21.15.57:6789/0 conn(0x7f47480a9d70 legacy=0x7f474806de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:05.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.610+0000 7f474d596640 1 -- 172.21.15.57:0/657356260 shutdown_connections 2024-03-29T10:49:05.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.610+0000 7f474d596640 1 -- 172.21.15.57:0/657356260 >> 172.21.15.57:0/657356260 conn(0x7f4748066a00 msgr2=0x7f474813e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:05.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.610+0000 7f474d596640 1 -- 172.21.15.57:0/657356260 shutdown_connections 2024-03-29T10:49:05.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.610+0000 7f474d596640 1 -- 172.21.15.57:0/657356260 wait complete. 2024-03-29T10:49:05.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:05.611+0000 7f474d596640 1 librados: shutdown 2024-03-29T10:49:05.609 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:12.418 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:12.574 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.591+0000 7efff0ae4640 10 monclient: get_monmap_and_config 2024-03-29T10:49:12.574 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.591+0000 7efff0ae4640 10 monclient: build_initial_monmap 2024-03-29T10:49:12.574 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.591+0000 7efff0ae4640 10 monclient: monmap: 2024-03-29T10:49:12.574 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:49:12.574 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:12.574 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:49:12.592869+0000 2024-03-29T10:49:12.574 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:49:12.592869+0000 2024-03-29T10:49:12.574 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:12.574 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:12.574 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:49:12.574 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:49:12.575 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:49:12.575 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:12.575 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.592+0000 7efff0ae4640 1 Processor -- start 2024-03-29T10:49:12.576 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.592+0000 7efff0ae4640 1 -- start start 2024-03-29T10:49:12.576 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.592+0000 7efff0ae4640 10 monclient: init 2024-03-29T10:49:12.579 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.593+0000 7efff0ae4640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:12.579 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.593+0000 7efff0ae4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:49:12.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.596+0000 7efff0ae4640 10 monclient(hunting): picked mon.noname-a con 0x7effe4038a00 addr v1:172.21.15.57:6789/0 2024-03-29T10:49:12.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.597+0000 7efff0ae4640 10 monclient(hunting): picked mon.noname-b con 0x7effe4038de0 addr v1:172.21.15.57:6790/0 2024-03-29T10:49:12.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.597+0000 7effea59c640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7effe4038a00 0x7effe4033540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:53871/0 (socket says 172.21.15.57:53871) 2024-03-29T10:49:12.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.597+0000 7efff0ae4640 10 monclient(hunting): picked mon.noname-c con 0x7effe4030680 addr v1:172.21.15.179:6789/0 2024-03-29T10:49:12.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.597+0000 7efff0ae4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe40c54e0 con 0x7effe4038a00 2024-03-29T10:49:12.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.598+0000 7efff0ae4640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe4036450 con 0x7effe4030680 2024-03-29T10:49:12.581 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.598+0000 7efff0ae4640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe4039860 con 0x7effe4038de0 2024-03-29T10:49:12.581 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.598+0000 7efff0ae4640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:12.581 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.598+0000 7efff0ae4640 10 monclient(hunting): authenticate will time out at 1504.775059s 2024-03-29T10:49:12.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.601+0000 7effea59c640 1 -- 172.21.15.57:0/208830019 learned_addr learned my addr 172.21.15.57:0/208830019 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:49:12.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.601+0000 7effe959a640 1 -- 172.21.15.57:0/208830019 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404909205 0 0) 0x7effe40c54e0 con 0x7effe4038a00 2024-03-29T10:49:12.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.601+0000 7effe959a640 10 monclient(hunting): _negotiate 2024-03-29T10:49:12.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.601+0000 7effe959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:12.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.601+0000 7effe959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:12.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.601+0000 7effe959a640 10 monclient(hunting): my global_id is 4551 2024-03-29T10:49:12.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.601+0000 7effe959a640 1 -- 172.21.15.57:0/208830019 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effd4003520 con 0x7effe4038a00 2024-03-29T10:49:12.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7effe959a640 1 -- 172.21.15.57:0/208830019 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1157058632 0 0) 0x7effd4003520 con 0x7effe4038a00 2024-03-29T10:49:12.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7effe959a640 1 -- 172.21.15.57:0/208830019 >> v1:172.21.15.57:6790/0 conn(0x7effe4038de0 legacy=0x7effe40cadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:49:12.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7effe959a640 1 -- 172.21.15.57:0/208830019 >> v1:172.21.15.179:6789/0 conn(0x7effe4030680 legacy=0x7effe40d3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:12.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7effe959a640 10 monclient: _finish_hunting 0 2024-03-29T10:49:12.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7effe959a640 1 monclient: found mon.noname-a 2024-03-29T10:49:12.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7effe959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:12.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7effe959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:49:12.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7effe959a640 1 -- 172.21.15.57:0/208830019 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe40356b0 con 0x7effe4038a00 2024-03-29T10:49:12.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7effe959a640 10 monclient: _finish_auth 0 2024-03-29T10:49:12.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7effe959a640 10 monclient: _check_auth_tickets 2024-03-29T10:49:12.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7effe959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:12.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7effe959a640 1 -- 172.21.15.57:0/208830019 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7effdc001690 con 0x7effe4038a00 2024-03-29T10:49:12.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7effe959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:12.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7effe959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:12.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7effe959a640 10 monclient: dump: 2024-03-29T10:49:12.590 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:12.591 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:12.591 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:12.591 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:12.591 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:12.591 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:12.591 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:12.591 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:12.591 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:12.591 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:12.591 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7efff0ae4640 5 monclient: authenticate success, global_id 4551 2024-03-29T10:49:12.591 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7efff0ae4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:12.591 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7effe959a640 1 -- 172.21.15.57:0/208830019 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7effdc001860 con 0x7effe4038a00 2024-03-29T10:49:12.591 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7effe959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:12.591 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7efff0ae4640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:12.591 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7efff0ae4640 4 set_mon_vals no callback set 2024-03-29T10:49:12.592 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7effe959a640 1 -- 172.21.15.57:0/208830019 <== mon.0 v1:172.21.15.57:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7effdc001c70 con 0x7effe4038a00 2024-03-29T10:49:12.592 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7effe959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:12.592 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7effe959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:12.592 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.602+0000 7effe959a640 10 monclient: dump: 2024-03-29T10:49:12.592 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:12.592 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:12.592 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:12.592 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:12.592 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:12.592 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:12.592 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:12.592 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:12.592 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:12.592 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:12.592 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.603+0000 7efff0ae4640 10 monclient: shutdown 2024-03-29T10:49:12.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.603+0000 7efff0ae4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:12.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.603+0000 7efff0ae4640 1 -- 172.21.15.57:0/208830019 >> v1:172.21.15.57:6789/0 conn(0x7effe4038a00 legacy=0x7effe4033540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:12.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.603+0000 7efff0ae4640 1 -- 172.21.15.57:0/208830019 shutdown_connections 2024-03-29T10:49:12.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.603+0000 7efff0ae4640 1 -- 172.21.15.57:0/208830019 >> 172.21.15.57:0/208830019 conn(0x7effe4005c20 msgr2=0x7effe4006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:12.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.603+0000 7efff0ae4640 1 -- 172.21.15.57:0/208830019 shutdown_connections 2024-03-29T10:49:12.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.603+0000 7efff0ae4640 1 -- 172.21.15.57:0/208830019 wait complete. 2024-03-29T10:49:12.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.605+0000 7efff0ae4640 10 monclient: build_initial_monmap 2024-03-29T10:49:12.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.605+0000 7efff0ae4640 10 monclient: monmap: 2024-03-29T10:49:12.593 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:49:12.593 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:12.593 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:49:12.593 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:49:12.593 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:12.593 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:12.593 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:49:12.594 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:49:12.594 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:49:12.594 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:12.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.605+0000 7efff0ae4640 1 librados: starting msgr at 2024-03-29T10:49:12.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.605+0000 7efff0ae4640 1 librados: starting objecter 2024-03-29T10:49:12.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.606+0000 7efff0ae4640 1 Processor -- start 2024-03-29T10:49:12.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.606+0000 7efff0ae4640 1 -- start start 2024-03-29T10:49:12.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.606+0000 7efff0ae4640 1 librados: setting wanted keys 2024-03-29T10:49:12.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.606+0000 7efff0ae4640 1 librados: calling monclient init 2024-03-29T10:49:12.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.606+0000 7efff0ae4640 10 monclient: init 2024-03-29T10:49:12.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.607+0000 7efff0ae4640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:12.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.607+0000 7efff0ae4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:49:12.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.609+0000 7efff0ae4640 10 monclient(hunting): picked mon.noname-a con 0x7effe4030680 addr v1:172.21.15.57:6789/0 2024-03-29T10:49:12.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.611+0000 7effea59c640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7effe4030680 0x7effe402f9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:54097/0 (socket says 172.21.15.57:54097) 2024-03-29T10:49:12.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.611+0000 7efff0ae4640 10 monclient(hunting): picked mon.noname-b con 0x7effe4038a00 addr v1:172.21.15.179:6789/0 2024-03-29T10:49:12.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.612+0000 7efff0ae4640 10 monclient(hunting): picked mon.noname-c con 0x7effe402dcf0 addr v1:172.21.15.57:6790/0 2024-03-29T10:49:12.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.612+0000 7effe9d9b640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7effe4038a00 0x7effe415fdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:44947/0 (socket says 172.21.15.57:44947) 2024-03-29T10:49:12.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.612+0000 7effe9d9b640 1 -- 172.21.15.57:0/455299610 learned_addr learned my addr 172.21.15.57:0/455299610 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:49:12.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.612+0000 7efff0ae4640 1 -- 172.21.15.57:0/455299610 --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe40c10a0 con 0x7effe4030680 2024-03-29T10:49:12.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.612+0000 7efff0ae4640 1 -- 172.21.15.57:0/455299610 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe4030100 con 0x7effe4038a00 2024-03-29T10:49:12.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.612+0000 7efff0ae4640 1 -- 172.21.15.57:0/455299610 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe402ef50 con 0x7effe402dcf0 2024-03-29T10:49:12.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.612+0000 7efff0ae4640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:12.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.612+0000 7efff0ae4640 10 monclient(hunting): authenticate will time out at 1504.789745s 2024-03-29T10:49:12.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.614+0000 7effdaffd640 10 client.?.objecter ms_handle_connect 0x7effe4038a00 2024-03-29T10:49:12.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.614+0000 7effdaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:12.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.614+0000 7effdaffd640 10 client.?.objecter ms_handle_connect 0x7effe4030680 2024-03-29T10:49:12.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.614+0000 7effdaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:12.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.614+0000 7effdaffd640 10 client.?.objecter ms_handle_connect 0x7effe402dcf0 2024-03-29T10:49:12.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.614+0000 7effdaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:12.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.614+0000 7effdaffd640 1 -- 172.21.15.57:0/455299610 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431007933 0 0) 0x7effe402ef50 con 0x7effe402dcf0 2024-03-29T10:49:12.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.614+0000 7effdaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:12.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.614+0000 7effdaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:12.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.614+0000 7effdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:12.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.614+0000 7effdaffd640 10 monclient(hunting): my global_id is 4610 2024-03-29T10:49:12.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.614+0000 7effdaffd640 1 -- 172.21.15.57:0/455299610 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effcc004eb0 con 0x7effe402dcf0 2024-03-29T10:49:12.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.615+0000 7effdaffd640 1 -- 172.21.15.57:0/455299610 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539443817 0 0) 0x7effe4030100 con 0x7effe4038a00 2024-03-29T10:49:12.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.615+0000 7effdaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:12.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.615+0000 7effdaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:12.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.615+0000 7effdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:12.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.615+0000 7effdaffd640 10 monclient(hunting): my global_id is 4555 2024-03-29T10:49:12.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.615+0000 7effdaffd640 1 -- 172.21.15.57:0/455299610 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effe402ef50 con 0x7effe4038a00 2024-03-29T10:49:12.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.615+0000 7effdaffd640 1 -- 172.21.15.57:0/455299610 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4197987061 0 0) 0x7effcc004eb0 con 0x7effe402dcf0 2024-03-29T10:49:12.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.615+0000 7effdaffd640 1 -- 172.21.15.57:0/455299610 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effe4030100 con 0x7effe402dcf0 2024-03-29T10:49:12.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.615+0000 7effdaffd640 1 -- 172.21.15.57:0/455299610 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7effec04fdd0 con 0x7effe402dcf0 2024-03-29T10:49:12.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.615+0000 7effdaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:12.609 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.615+0000 7effdaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:12.609 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.615+0000 7effdaffd640 10 monclient(hunting): dump: 2024-03-29T10:49:12.609 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:12.609 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:12.609 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:12.609 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:12.609 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:12.609 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:12.609 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:12.609 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:12.609 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:12.609 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:12.609 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.615+0000 7effdaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:12.609 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.615+0000 7effdaffd640 1 -- 172.21.15.57:0/455299610 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2473385091 0 0) 0x7effe40c10a0 con 0x7effe4030680 2024-03-29T10:49:12.609 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.615+0000 7effdaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:12.610 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.615+0000 7effdaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:12.610 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.615+0000 7effdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:12.610 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.615+0000 7effdaffd640 10 monclient(hunting): my global_id is 4554 2024-03-29T10:49:12.610 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.615+0000 7effdaffd640 1 -- 172.21.15.57:0/455299610 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effcc004eb0 con 0x7effe4030680 2024-03-29T10:49:12.610 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.615+0000 7effdaffd640 1 -- 172.21.15.57:0/455299610 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1648677906 0 0) 0x7effe402ef50 con 0x7effe4038a00 2024-03-29T10:49:12.610 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.615+0000 7effdaffd640 1 -- 172.21.15.57:0/455299610 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effe40c10a0 con 0x7effe4038a00 2024-03-29T10:49:12.610 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.616+0000 7effdaffd640 1 -- 172.21.15.57:0/455299610 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7effe0001750 con 0x7effe4038a00 2024-03-29T10:49:12.610 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.616+0000 7effdaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:12.610 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.616+0000 7effdaffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:12.610 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.616+0000 7effdaffd640 10 monclient(hunting): dump: 2024-03-29T10:49:12.610 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:12.610 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:12.611 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:12.611 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:12.611 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:12.611 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:12.611 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:12.611 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:12.611 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:12.611 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:12.611 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.616+0000 7effdaffd640 1 -- 172.21.15.57:0/455299610 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3247759624 0 0) 0x7effe4030100 con 0x7effe402dcf0 2024-03-29T10:49:12.611 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.616+0000 7effdaffd640 1 -- 172.21.15.57:0/455299610 >> v1:172.21.15.179:6789/0 conn(0x7effe4038a00 legacy=0x7effe415fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:12.611 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.616+0000 7effdaffd640 1 -- 172.21.15.57:0/455299610 >> v1:172.21.15.57:6789/0 conn(0x7effe4030680 legacy=0x7effe402f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:12.611 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.616+0000 7effdaffd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:12.611 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.616+0000 7effdaffd640 1 monclient: found mon.c 2024-03-29T10:49:12.611 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.616+0000 7effdaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:12.611 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.616+0000 7effdaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:49:12.612 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.616+0000 7effdaffd640 1 -- 172.21.15.57:0/455299610 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe4160be0 con 0x7effe402dcf0 2024-03-29T10:49:12.612 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.616+0000 7effdaffd640 10 monclient: _finish_auth 0 2024-03-29T10:49:12.612 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.616+0000 7effdaffd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:12.612 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.616+0000 7effdaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:12.612 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.616+0000 7efff0ae4640 5 monclient: authenticate success, global_id 4610 2024-03-29T10:49:12.612 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.616+0000 7efff0ae4640 10 monclient: _renew_subs 2024-03-29T10:49:12.612 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.616+0000 7efff0ae4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:49:12.612 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.616+0000 7efff0ae4640 1 -- 172.21.15.57:0/455299610 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe40c8290 con 0x7effe402dcf0 2024-03-29T10:49:12.612 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.617+0000 7efff0ae4640 10 client.4610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:12.612 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.617+0000 7efff0ae4640 10 monclient: _renew_subs 2024-03-29T10:49:12.613 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.617+0000 7efff0ae4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:49:12.613 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.617+0000 7efff0ae4640 1 -- 172.21.15.57:0/455299610 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7effe402d620 con 0x7effe402dcf0 2024-03-29T10:49:12.613 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.617+0000 7efff0ae4640 1 librados: init done 2024-03-29T10:49:12.613 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.618+0000 7efff0ae4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:12.613 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.618+0000 7efff0ae4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:12.613 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.618+0000 7efff0ae4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:49:12.613 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.618+0000 7efff0ae4640 1 -- 172.21.15.57:0/455299610 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe4007bb0 con 0x7effe402dcf0 2024-03-29T10:49:12.613 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.619+0000 7effdaffd640 1 -- 172.21.15.57:0/455299610 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7effec04f400 con 0x7effe402dcf0 2024-03-29T10:49:12.613 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.619+0000 7effdaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:12.613 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.619+0000 7effdaffd640 1 -- 172.21.15.57:0/455299610 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7effec075710 con 0x7effe402dcf0 2024-03-29T10:49:12.613 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.619+0000 7effdaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:12.613 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.619+0000 7effdaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:12.613 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.619+0000 7effdaffd640 10 monclient: dump: 2024-03-29T10:49:12.614 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:12.614 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:12.614 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:12.614 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:12.614 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:12.614 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:12.614 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:12.614 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:12.614 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:12.614 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:12.614 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.619+0000 7effdaffd640 1 -- 172.21.15.57:0/455299610 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7effec067bf0 con 0x7effe402dcf0 2024-03-29T10:49:12.615 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.620+0000 7effdaffd640 1 -- 172.21.15.57:0/455299610 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7effec0fe250 con 0x7effe402dcf0 2024-03-29T10:49:12.615 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.620+0000 7effdaffd640 10 client.4610.objecter ms_dispatch 0x7effec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:12.615 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.620+0000 7effdaffd640 3 client.4610.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:12.615 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.620+0000 7effdaffd640 3 client.4610.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:12.615 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.620+0000 7effdaffd640 20 client.4610.objecter dump_active .. 0 homeless 2024-03-29T10:49:12.615 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.621+0000 7effe8d99640 4 set_mon_vals no callback set 2024-03-29T10:49:12.615 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.622+0000 7effea59c640 10 monclient: get_auth_request con 0x7effcc06ddf0 auth_method 0 2024-03-29T10:49:12.615 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.622+0000 7effdaffd640 10 client.4610.objecter ms_handle_connect 0x7effcc06ddf0 2024-03-29T10:49:12.617 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.630+0000 7effdaffd640 1 -- 172.21.15.57:0/455299610 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7effec0c9bb0 con 0x7effe402dcf0 2024-03-29T10:49:12.617 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.630+0000 7effdaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:12.618 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.630+0000 7effdaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:12.942 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.960+0000 7efff0ae4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:12.943 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.960+0000 7efff0ae4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:12.943 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.960+0000 7efff0ae4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:49:12.943 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.960+0000 7efff0ae4640 1 -- 172.21.15.57:0/455299610 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7effe4005140 con 0x7effe402dcf0 2024-03-29T10:49:12.944 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.961+0000 7effdaffd640 1 -- 172.21.15.57:0/455299610 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7effec0cbae0 con 0x7effe402dcf0 2024-03-29T10:49:12.944 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.961+0000 7effdaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:12.944 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.961+0000 7effdaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:12.946 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:49:12.946 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:12.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.966+0000 7efff0ae4640 10 librados: watch_flush enter 2024-03-29T10:49:12.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.966+0000 7efff0ae4640 10 librados: watch_flush exit 2024-03-29T10:49:12.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.966+0000 7efff0ae4640 20 client.4610.objecter shutdown clearing up homeless session... 2024-03-29T10:49:12.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.966+0000 7efff0ae4640 10 client.4610.objecter successfully canceled tick 2024-03-29T10:49:12.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.967+0000 7efff0ae4640 1 -- 172.21.15.57:0/455299610 >> v1:172.21.15.179:6816/2698295936 conn(0x7effcc06ddf0 legacy=0x7effcc0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:12.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.967+0000 7efff0ae4640 10 monclient: shutdown 2024-03-29T10:49:12.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.967+0000 7efff0ae4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:12.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.967+0000 7efff0ae4640 1 -- 172.21.15.57:0/455299610 >> v1:172.21.15.57:6790/0 conn(0x7effe402dcf0 legacy=0x7effe41604d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:12.951 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.968+0000 7efff0ae4640 1 -- 172.21.15.57:0/455299610 shutdown_connections 2024-03-29T10:49:12.951 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.968+0000 7efff0ae4640 1 -- 172.21.15.57:0/455299610 >> 172.21.15.57:0/455299610 conn(0x7effe4005c20 msgr2=0x7effe40340b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:12.951 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.968+0000 7efff0ae4640 1 -- 172.21.15.57:0/455299610 shutdown_connections 2024-03-29T10:49:12.951 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.968+0000 7efff0ae4640 1 -- 172.21.15.57:0/455299610 wait complete. 2024-03-29T10:49:12.951 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:12.968+0000 7efff0ae4640 1 librados: shutdown 2024-03-29T10:49:12.970 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:19.777 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:19.923 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.940+0000 7f642ee84640 10 monclient: get_monmap_and_config 2024-03-29T10:49:19.924 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.940+0000 7f642ee84640 10 monclient: build_initial_monmap 2024-03-29T10:49:19.924 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.940+0000 7f642ee84640 10 monclient: monmap: 2024-03-29T10:49:19.924 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:49:19.924 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:19.924 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:49:19.942220+0000 2024-03-29T10:49:19.924 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:49:19.942220+0000 2024-03-29T10:49:19.924 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:19.924 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:19.924 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:49:19.924 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:49:19.924 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:49:19.924 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:19.926 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.941+0000 7f642ee84640 1 Processor -- start 2024-03-29T10:49:19.926 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.941+0000 7f642ee84640 1 -- start start 2024-03-29T10:49:19.926 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.941+0000 7f642ee84640 10 monclient: init 2024-03-29T10:49:19.926 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.942+0000 7f642ee84640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:19.926 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.942+0000 7f642ee84640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:49:19.926 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.942+0000 7f642ee84640 10 monclient(hunting): picked mon.noname-c con 0x7f64280b0770 addr v1:172.21.15.179:6789/0 2024-03-29T10:49:19.926 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.942+0000 7f642ee84640 10 monclient(hunting): picked mon.noname-b con 0x7f642814a700 addr v1:172.21.15.57:6790/0 2024-03-29T10:49:19.926 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.942+0000 7f642ee84640 10 monclient(hunting): picked mon.noname-a con 0x7f64280ac5a0 addr v1:172.21.15.57:6789/0 2024-03-29T10:49:19.927 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.942+0000 7f642ee84640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64280aba70 con 0x7f64280ac5a0 2024-03-29T10:49:19.927 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.942+0000 7f642ee84640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64280aecb0 con 0x7f64280b0770 2024-03-29T10:49:19.927 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.942+0000 7f642ee84640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64280af150 con 0x7f642814a700 2024-03-29T10:49:19.927 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.942+0000 7f642ee84640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:19.927 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.942+0000 7f642ee84640 10 monclient(hunting): authenticate will time out at 1512.119559s 2024-03-29T10:49:19.929 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.946+0000 7f6427fff640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f642814a700 0x7f64280abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:34244/0 (socket says 172.21.15.57:34244) 2024-03-29T10:49:19.929 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.946+0000 7f6427fff640 1 -- 172.21.15.57:0/1029982397 learned_addr learned my addr 172.21.15.57:0/1029982397 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:49:19.929 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.947+0000 7f642d421640 1 --1- 172.21.15.57:0/1029982397 >> v1:172.21.15.57:6789/0 conn(0x7f64280ac5a0 0x7f6428151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:60193/0 (socket says 172.21.15.57:60193) 2024-03-29T10:49:19.935 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.947+0000 7f64277fe640 1 -- 172.21.15.57:0/1029982397 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45967245 0 0) 0x7f64280aecb0 con 0x7f64280b0770 2024-03-29T10:49:19.935 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.947+0000 7f64277fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:19.935 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.947+0000 7f64277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:19.935 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.947+0000 7f64277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:19.935 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.947+0000 7f64277fe640 10 monclient(hunting): my global_id is 4558 2024-03-29T10:49:19.935 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 1 -- 172.21.15.57:0/1029982397 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6410003520 con 0x7f64280b0770 2024-03-29T10:49:19.935 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 1 -- 172.21.15.57:0/1029982397 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3520794260 0 0) 0x7f64280aba70 con 0x7f64280ac5a0 2024-03-29T10:49:19.935 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:19.935 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:19.935 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:19.935 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 10 monclient(hunting): my global_id is 4557 2024-03-29T10:49:19.936 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 1 -- 172.21.15.57:0/1029982397 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64280aecb0 con 0x7f64280ac5a0 2024-03-29T10:49:19.936 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 1 -- 172.21.15.57:0/1029982397 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3534908558 0 0) 0x7f64280aecb0 con 0x7f64280ac5a0 2024-03-29T10:49:19.936 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 1 -- 172.21.15.57:0/1029982397 >> v1:172.21.15.57:6790/0 conn(0x7f642814a700 legacy=0x7f64280abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:19.936 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 1 -- 172.21.15.57:0/1029982397 >> v1:172.21.15.179:6789/0 conn(0x7f64280b0770 legacy=0x7f6428149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:19.936 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 10 monclient: _finish_hunting 0 2024-03-29T10:49:19.936 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 1 monclient: found mon.noname-a 2024-03-29T10:49:19.936 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:19.936 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:49:19.936 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 1 -- 172.21.15.57:0/1029982397 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64280ac9d0 con 0x7f64280ac5a0 2024-03-29T10:49:19.936 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 10 monclient: _finish_auth 0 2024-03-29T10:49:19.936 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 10 monclient: _check_auth_tickets 2024-03-29T10:49:19.936 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:19.936 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 1 -- 172.21.15.57:0/1029982397 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f64200019f0 con 0x7f64280ac5a0 2024-03-29T10:49:19.936 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:19.937 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:19.937 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 10 monclient: dump: 2024-03-29T10:49:19.937 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:19.937 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:19.937 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:19.937 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:19.937 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:19.937 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:19.937 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:19.937 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:19.937 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:19.937 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:19.937 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f642ee84640 5 monclient: authenticate success, global_id 4557 2024-03-29T10:49:19.937 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f642ee84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:19.938 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 1 -- 172.21.15.57:0/1029982397 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6420001b50 con 0x7f64280ac5a0 2024-03-29T10:49:19.938 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:19.938 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 1 -- 172.21.15.57:0/1029982397 <== mon.0 v1:172.21.15.57:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f6420001fa0 con 0x7f64280ac5a0 2024-03-29T10:49:19.938 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:19.938 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:19.938 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f64277fe640 10 monclient: dump: 2024-03-29T10:49:19.938 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:19.938 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:19.938 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:19.938 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:19.938 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:19.938 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:19.938 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:19.938 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:19.939 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:19.939 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:19.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f642ee84640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:19.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.948+0000 7f642ee84640 4 set_mon_vals no callback set 2024-03-29T10:49:19.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.949+0000 7f642ee84640 10 monclient: shutdown 2024-03-29T10:49:19.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.949+0000 7f642ee84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:19.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.949+0000 7f642ee84640 1 -- 172.21.15.57:0/1029982397 >> v1:172.21.15.57:6789/0 conn(0x7f64280ac5a0 legacy=0x7f6428151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:19.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.950+0000 7f642ee84640 1 -- 172.21.15.57:0/1029982397 shutdown_connections 2024-03-29T10:49:19.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.950+0000 7f642ee84640 1 -- 172.21.15.57:0/1029982397 >> 172.21.15.57:0/1029982397 conn(0x7f642806b660 msgr2=0x7f642806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:19.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.950+0000 7f642ee84640 1 -- 172.21.15.57:0/1029982397 shutdown_connections 2024-03-29T10:49:19.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.950+0000 7f642ee84640 1 -- 172.21.15.57:0/1029982397 wait complete. 2024-03-29T10:49:19.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.951+0000 7f642ee84640 10 monclient: build_initial_monmap 2024-03-29T10:49:19.939 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.951+0000 7f642ee84640 10 monclient: monmap: 2024-03-29T10:49:19.939 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:49:19.940 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:19.940 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:49:19.940 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:49:19.940 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:19.940 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:19.940 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:49:19.940 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:49:19.940 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:49:19.940 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:19.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.951+0000 7f642ee84640 1 librados: starting msgr at 2024-03-29T10:49:19.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.951+0000 7f642ee84640 1 librados: starting objecter 2024-03-29T10:49:19.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.952+0000 7f642ee84640 1 Processor -- start 2024-03-29T10:49:19.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.952+0000 7f642ee84640 1 -- start start 2024-03-29T10:49:19.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.952+0000 7f642ee84640 1 librados: setting wanted keys 2024-03-29T10:49:19.940 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.952+0000 7f642ee84640 1 librados: calling monclient init 2024-03-29T10:49:19.941 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.952+0000 7f642ee84640 10 monclient: init 2024-03-29T10:49:19.946 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.953+0000 7f642ee84640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:19.946 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.953+0000 7f642ee84640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:49:19.946 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.953+0000 7f642ee84640 10 monclient(hunting): picked mon.noname-b con 0x7f64280ac5a0 addr v1:172.21.15.179:6789/0 2024-03-29T10:49:19.946 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.953+0000 7f642ee84640 10 monclient(hunting): picked mon.noname-a con 0x7f64280b0770 addr v1:172.21.15.57:6789/0 2024-03-29T10:49:19.946 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.953+0000 7f642ee84640 10 monclient(hunting): picked mon.noname-c con 0x7f642814a700 addr v1:172.21.15.57:6790/0 2024-03-29T10:49:19.946 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.953+0000 7f642ee84640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64281280d0 con 0x7f64280b0770 2024-03-29T10:49:19.946 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.953+0000 7f642ee84640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64281db600 con 0x7f64280ac5a0 2024-03-29T10:49:19.946 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.953+0000 7f642ee84640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64281239d0 con 0x7f642814a700 2024-03-29T10:49:19.946 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.953+0000 7f642ee84640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:19.947 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.953+0000 7f642ee84640 10 monclient(hunting): authenticate will time out at 1512.130938s 2024-03-29T10:49:19.947 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.956+0000 7f6427fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f64280b0770 0x7f64281dba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:60435/0 (socket says 172.21.15.57:60435) 2024-03-29T10:49:19.947 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.956+0000 7f6427fff640 1 -- 172.21.15.57:0/873522673 learned_addr learned my addr 172.21.15.57:0/873522673 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:49:19.947 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.956+0000 7f64257fa640 10 client.?.objecter ms_handle_connect 0x7f64280b0770 2024-03-29T10:49:19.947 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.956+0000 7f64257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:19.947 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.957+0000 7f64257fa640 1 -- 172.21.15.57:0/873522673 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130707570 0 0) 0x7f64281280d0 con 0x7f64280b0770 2024-03-29T10:49:19.947 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.957+0000 7f64257fa640 10 monclient(hunting): _negotiate 2024-03-29T10:49:19.947 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.957+0000 7f64257fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:19.947 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.957+0000 7f64257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:19.947 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.957+0000 7f64257fa640 10 monclient(hunting): my global_id is 4560 2024-03-29T10:49:19.947 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.957+0000 7f64257fa640 1 -- 172.21.15.57:0/873522673 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6414004eb0 con 0x7f64280b0770 2024-03-29T10:49:19.947 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.957+0000 7f64257fa640 10 client.?.objecter ms_handle_connect 0x7f642814a700 2024-03-29T10:49:19.947 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.957+0000 7f64257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:19.947 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.957+0000 7f64257fa640 1 -- 172.21.15.57:0/873522673 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2730004094 0 0) 0x7f6414004eb0 con 0x7f64280b0770 2024-03-29T10:49:19.947 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.957+0000 7f64257fa640 1 -- 172.21.15.57:0/873522673 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6414005300 con 0x7f64280b0770 2024-03-29T10:49:19.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.957+0000 7f64257fa640 1 -- 172.21.15.57:0/873522673 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f641c001890 con 0x7f64280b0770 2024-03-29T10:49:19.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.957+0000 7f64257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:19.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.957+0000 7f64257fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:19.948 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.957+0000 7f64257fa640 10 monclient(hunting): dump: 2024-03-29T10:49:19.948 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:19.948 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:19.948 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:19.948 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:19.948 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:19.948 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:19.948 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:19.948 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:19.948 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:19.948 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:19.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.957+0000 7f64257fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:19.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f64257fa640 1 -- 172.21.15.57:0/873522673 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1757363701 0 0) 0x7f6414005300 con 0x7f64280b0770 2024-03-29T10:49:19.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f64257fa640 1 -- 172.21.15.57:0/873522673 >> v1:172.21.15.57:6790/0 conn(0x7f642814a700 legacy=0x7f64281e6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:19.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f64257fa640 1 -- 172.21.15.57:0/873522673 >> v1:172.21.15.179:6789/0 conn(0x7f64280ac5a0 legacy=0x7f64281451c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:19.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f64257fa640 10 monclient: _finish_hunting 0 2024-03-29T10:49:19.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f64257fa640 1 monclient: found mon.a 2024-03-29T10:49:19.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f64257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:19.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f64257fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:49:19.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f64257fa640 1 -- 172.21.15.57:0/873522673 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6428128320 con 0x7f64280b0770 2024-03-29T10:49:19.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f64257fa640 10 monclient: _finish_auth 0 2024-03-29T10:49:19.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f64257fa640 10 monclient: _check_auth_tickets 2024-03-29T10:49:19.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f64257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:19.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f642ee84640 5 monclient: authenticate success, global_id 4560 2024-03-29T10:49:19.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f642ee84640 10 monclient: _renew_subs 2024-03-29T10:49:19.949 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f642ee84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:49:19.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f642ee84640 1 -- 172.21.15.57:0/873522673 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6428125f30 con 0x7f64280b0770 2024-03-29T10:49:19.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f642ee84640 10 client.4560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:19.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f642ee84640 10 monclient: _renew_subs 2024-03-29T10:49:19.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f642ee84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:49:19.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f642ee84640 1 -- 172.21.15.57:0/873522673 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6428126450 con 0x7f64280b0770 2024-03-29T10:49:19.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f642ee84640 1 librados: init done 2024-03-29T10:49:19.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f64257fa640 1 -- 172.21.15.57:0/873522673 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f641c0047e0 con 0x7f64280b0770 2024-03-29T10:49:19.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f64257fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:19.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f64257fa640 1 -- 172.21.15.57:0/873522673 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f641c004bb0 con 0x7f64280b0770 2024-03-29T10:49:19.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f64257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:19.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f64257fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:19.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.958+0000 7f64257fa640 10 monclient: dump: 2024-03-29T10:49:19.950 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:19.950 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:19.951 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:19.951 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:19.951 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:19.951 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:19.951 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:19.951 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:19.951 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:19.951 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:19.951 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.960+0000 7f6426ffd640 4 set_mon_vals no callback set 2024-03-29T10:49:19.951 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.960+0000 7f642ee84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:19.951 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.961+0000 7f642ee84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:19.951 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.961+0000 7f642ee84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:49:19.951 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.961+0000 7f642ee84640 1 -- 172.21.15.57:0/873522673 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64280ab560 con 0x7f64280b0770 2024-03-29T10:49:19.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.964+0000 7f64257fa640 1 -- 172.21.15.57:0/873522673 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f641c004d90 con 0x7f64280b0770 2024-03-29T10:49:19.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.965+0000 7f642cc20640 10 monclient: get_auth_request con 0x7f641406de90 auth_method 0 2024-03-29T10:49:19.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.965+0000 7f64257fa640 1 -- 172.21.15.57:0/873522673 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f641c0049d0 con 0x7f64280b0770 2024-03-29T10:49:19.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.965+0000 7f64257fa640 10 client.4560.objecter ms_dispatch 0x7f6428000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:19.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.965+0000 7f64257fa640 3 client.4560.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:19.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.965+0000 7f64257fa640 3 client.4560.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:19.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.965+0000 7f64257fa640 20 client.4560.objecter dump_active .. 0 homeless 2024-03-29T10:49:19.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.965+0000 7f64257fa640 10 client.4560.objecter ms_handle_connect 0x7f641406de90 2024-03-29T10:49:19.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.970+0000 7f64257fa640 1 -- 172.21.15.57:0/873522673 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f641c050960 con 0x7f64280b0770 2024-03-29T10:49:19.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.970+0000 7f64257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:19.954 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:19.970+0000 7f64257fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:20.290 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:20.308+0000 7f642ee84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:20.291 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:20.308+0000 7f642ee84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:20.291 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:20.308+0000 7f642ee84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:49:20.291 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:20.308+0000 7f642ee84640 1 -- 172.21.15.57:0/873522673 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6428075f10 con 0x7f64280b0770 2024-03-29T10:49:20.293 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:49:20.293 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:20.295 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:20.308+0000 7f64257fa640 1 -- 172.21.15.57:0/873522673 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f641c050960 con 0x7f64280b0770 2024-03-29T10:49:20.295 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:20.308+0000 7f64257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:20.295 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:20.308+0000 7f64257fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:20.295 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:20.312+0000 7f642ee84640 10 librados: watch_flush enter 2024-03-29T10:49:20.295 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:20.312+0000 7f642ee84640 10 librados: watch_flush exit 2024-03-29T10:49:20.295 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:20.312+0000 7f642ee84640 20 client.4560.objecter shutdown clearing up homeless session... 2024-03-29T10:49:20.295 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:20.312+0000 7f642ee84640 10 client.4560.objecter successfully canceled tick 2024-03-29T10:49:20.295 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:20.312+0000 7f642ee84640 1 -- 172.21.15.57:0/873522673 >> v1:172.21.15.179:6816/2698295936 conn(0x7f641406de90 legacy=0x7f6414070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:20.295 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:20.312+0000 7f642ee84640 10 monclient: shutdown 2024-03-29T10:49:20.296 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:20.312+0000 7f642ee84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:20.296 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:20.312+0000 7f642ee84640 1 -- 172.21.15.57:0/873522673 >> v1:172.21.15.57:6789/0 conn(0x7f64280b0770 legacy=0x7f64281dba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:20.296 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:20.313+0000 7f642ee84640 1 -- 172.21.15.57:0/873522673 shutdown_connections 2024-03-29T10:49:20.296 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:20.313+0000 7f642ee84640 1 -- 172.21.15.57:0/873522673 >> 172.21.15.57:0/873522673 conn(0x7f642806b660 msgr2=0x7f64280b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:20.296 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:20.313+0000 7f642ee84640 1 -- 172.21.15.57:0/873522673 shutdown_connections 2024-03-29T10:49:20.296 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:20.313+0000 7f642ee84640 1 -- 172.21.15.57:0/873522673 wait complete. 2024-03-29T10:49:20.297 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:20.313+0000 7f642ee84640 1 librados: shutdown 2024-03-29T10:49:20.313 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:27.121 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:27.270 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.287+0000 7f88c2515640 10 monclient: get_monmap_and_config 2024-03-29T10:49:27.271 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.287+0000 7f88c2515640 10 monclient: build_initial_monmap 2024-03-29T10:49:27.271 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.287+0000 7f88c2515640 10 monclient: monmap: 2024-03-29T10:49:27.271 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:49:27.271 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:27.271 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:49:27.289475+0000 2024-03-29T10:49:27.271 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:49:27.289475+0000 2024-03-29T10:49:27.271 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:27.271 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:27.271 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:49:27.271 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:49:27.271 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:49:27.272 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:27.272 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.289+0000 7f88c2515640 1 Processor -- start 2024-03-29T10:49:27.272 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.289+0000 7f88c2515640 1 -- start start 2024-03-29T10:49:27.272 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.289+0000 7f88c2515640 10 monclient: init 2024-03-29T10:49:27.273 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.290+0000 7f88c2515640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:27.273 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.290+0000 7f88c2515640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:27.273 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.290+0000 7f88c2515640 10 monclient(hunting): picked mon.noname-b con 0x7f88bc1457f0 addr v1:172.21.15.57:6790/0 2024-03-29T10:49:27.273 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.290+0000 7f88c2515640 10 monclient(hunting): picked mon.noname-c con 0x7f88bc13f650 addr v1:172.21.15.179:6789/0 2024-03-29T10:49:27.273 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.290+0000 7f88c2515640 10 monclient(hunting): picked mon.noname-a con 0x7f88bc0a9d70 addr v1:172.21.15.57:6789/0 2024-03-29T10:49:27.273 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.290+0000 7f88c2515640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88bc0aaef0 con 0x7f88bc0a9d70 2024-03-29T10:49:27.273 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.290+0000 7f88c2515640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88bc145bd0 con 0x7f88bc13f650 2024-03-29T10:49:27.273 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.290+0000 7f88c2515640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88bc145eb0 con 0x7f88bc1457f0 2024-03-29T10:49:27.274 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.290+0000 7f88c2515640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:27.274 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.290+0000 7f88c2515640 10 monclient(hunting): authenticate will time out at 1519.467434s 2024-03-29T10:49:27.274 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.290+0000 7f88c0ab2640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f88bc0a9d70 0x7f88bc14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:54818/0 (socket says 172.21.15.57:54818) 2024-03-29T10:49:27.274 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.290+0000 7f88c0ab2640 1 -- 172.21.15.57:0/2244149615 learned_addr learned my addr 172.21.15.57:0/2244149615 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:49:27.275 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.292+0000 7f88baffd640 1 -- 172.21.15.57:0/2244149615 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102111335 0 0) 0x7f88bc0aaef0 con 0x7f88bc0a9d70 2024-03-29T10:49:27.275 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.292+0000 7f88baffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:27.275 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.292+0000 7f88baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:27.275 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.292+0000 7f88baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:27.275 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.292+0000 7f88baffd640 10 monclient(hunting): my global_id is 4563 2024-03-29T10:49:27.275 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.292+0000 7f88baffd640 1 -- 172.21.15.57:0/2244149615 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88a4003520 con 0x7f88bc0a9d70 2024-03-29T10:49:27.275 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.292+0000 7f88baffd640 1 -- 172.21.15.57:0/2244149615 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 11512534 0 0) 0x7f88a4003520 con 0x7f88bc0a9d70 2024-03-29T10:49:27.275 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.292+0000 7f88baffd640 1 -- 172.21.15.57:0/2244149615 >> v1:172.21.15.57:6790/0 conn(0x7f88bc1457f0 legacy=0x7f88bc13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:27.275 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.293+0000 7f88baffd640 1 -- 172.21.15.57:0/2244149615 >> v1:172.21.15.179:6789/0 conn(0x7f88bc13f650 legacy=0x7f88bc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:27.276 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.293+0000 7f88baffd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:27.276 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.293+0000 7f88baffd640 1 monclient: found mon.noname-a 2024-03-29T10:49:27.276 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.293+0000 7f88baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:27.277 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.293+0000 7f88baffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:49:27.277 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.293+0000 7f88baffd640 1 -- 172.21.15.57:0/2244149615 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88bc146340 con 0x7f88bc0a9d70 2024-03-29T10:49:27.277 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.293+0000 7f88baffd640 10 monclient: _finish_auth 0 2024-03-29T10:49:27.277 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.293+0000 7f88baffd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:27.277 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.293+0000 7f88baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:27.277 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.293+0000 7f88baffd640 1 -- 172.21.15.57:0/2244149615 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f88b4001a10 con 0x7f88bc0a9d70 2024-03-29T10:49:27.277 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.293+0000 7f88baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:27.277 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.293+0000 7f88baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:27.277 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.293+0000 7f88baffd640 10 monclient: dump: 2024-03-29T10:49:27.277 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:27.277 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:27.277 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:27.277 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:27.277 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:27.278 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:27.278 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:27.278 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:27.278 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:27.278 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:27.278 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.293+0000 7f88c2515640 5 monclient: authenticate success, global_id 4563 2024-03-29T10:49:27.278 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.293+0000 7f88c2515640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:27.279 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.294+0000 7f88baffd640 1 -- 172.21.15.57:0/2244149615 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88b4001be0 con 0x7f88bc0a9d70 2024-03-29T10:49:27.279 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.294+0000 7f88baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:27.279 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.294+0000 7f88baffd640 1 -- 172.21.15.57:0/2244149615 <== mon.0 v1:172.21.15.57:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f88b4002030 con 0x7f88bc0a9d70 2024-03-29T10:49:27.279 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.294+0000 7f88baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:27.279 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.294+0000 7f88baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:27.280 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.294+0000 7f88baffd640 10 monclient: dump: 2024-03-29T10:49:27.280 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:27.280 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:27.280 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:27.280 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:27.280 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:27.280 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:27.280 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:27.280 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:27.280 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:27.280 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:27.280 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.294+0000 7f88c2515640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:27.280 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.294+0000 7f88c2515640 4 set_mon_vals no callback set 2024-03-29T10:49:27.280 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.295+0000 7f88c2515640 10 monclient: shutdown 2024-03-29T10:49:27.280 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.295+0000 7f88c2515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:27.281 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.295+0000 7f88c2515640 1 -- 172.21.15.57:0/2244149615 >> v1:172.21.15.57:6789/0 conn(0x7f88bc0a9d70 legacy=0x7f88bc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:27.281 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.295+0000 7f88c2515640 1 -- 172.21.15.57:0/2244149615 shutdown_connections 2024-03-29T10:49:27.281 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.295+0000 7f88c2515640 1 -- 172.21.15.57:0/2244149615 >> 172.21.15.57:0/2244149615 conn(0x7f88bc066a00 msgr2=0x7f88bc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:27.281 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.295+0000 7f88c2515640 1 -- 172.21.15.57:0/2244149615 shutdown_connections 2024-03-29T10:49:27.283 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.295+0000 7f88c2515640 1 -- 172.21.15.57:0/2244149615 wait complete. 2024-03-29T10:49:27.283 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.296+0000 7f88c2515640 10 monclient: build_initial_monmap 2024-03-29T10:49:27.283 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.296+0000 7f88c2515640 10 monclient: monmap: 2024-03-29T10:49:27.283 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:49:27.283 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:27.283 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:49:27.283 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:49:27.283 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:27.283 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:27.283 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:49:27.283 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:49:27.283 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:49:27.284 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:27.284 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.297+0000 7f88c2515640 1 librados: starting msgr at 2024-03-29T10:49:27.284 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.297+0000 7f88c2515640 1 librados: starting objecter 2024-03-29T10:49:27.284 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.297+0000 7f88c2515640 1 Processor -- start 2024-03-29T10:49:27.284 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.297+0000 7f88c2515640 1 -- start start 2024-03-29T10:49:27.284 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.297+0000 7f88c2515640 1 librados: setting wanted keys 2024-03-29T10:49:27.284 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.298+0000 7f88c2515640 1 librados: calling monclient init 2024-03-29T10:49:27.284 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.298+0000 7f88c2515640 10 monclient: init 2024-03-29T10:49:27.284 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.298+0000 7f88c2515640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:27.284 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.298+0000 7f88c2515640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:49:27.284 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.299+0000 7f88c2515640 10 monclient(hunting): picked mon.noname-b con 0x7f88bc0a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T10:49:27.284 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.299+0000 7f88bbfff640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7f88bc0a9d70 0x7f88bc1d7a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:49158/0 (socket says 172.21.15.57:49158) 2024-03-29T10:49:27.284 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.299+0000 7f88c2515640 10 monclient(hunting): picked mon.noname-a con 0x7f88bc13f650 addr v1:172.21.15.57:6789/0 2024-03-29T10:49:27.284 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.299+0000 7f88bbfff640 1 -- 172.21.15.57:0/3878891526 learned_addr learned my addr 172.21.15.57:0/3878891526 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:49:27.292 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.299+0000 7f88c2515640 10 monclient(hunting): picked mon.noname-c con 0x7f88bc1457f0 addr v1:172.21.15.57:6790/0 2024-03-29T10:49:27.292 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.301+0000 7f88c2515640 1 -- 172.21.15.57:0/3878891526 --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88bc1d7570 con 0x7f88bc13f650 2024-03-29T10:49:27.292 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.301+0000 7f88c2515640 1 -- 172.21.15.57:0/3878891526 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88bc13fda0 con 0x7f88bc0a9d70 2024-03-29T10:49:27.292 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.301+0000 7f88c2515640 1 -- 172.21.15.57:0/3878891526 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88bc0b1660 con 0x7f88bc1457f0 2024-03-29T10:49:27.292 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.301+0000 7f88c2515640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:27.292 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.301+0000 7f88c2515640 10 monclient(hunting): authenticate will time out at 1519.478104s 2024-03-29T10:49:27.292 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.301+0000 7f88b8ff9640 10 client.?.objecter ms_handle_connect 0x7f88bc13f650 2024-03-29T10:49:27.292 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.301+0000 7f88b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:27.292 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.301+0000 7f88b8ff9640 10 client.?.objecter ms_handle_connect 0x7f88bc0a9d70 2024-03-29T10:49:27.293 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.301+0000 7f88b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:27.293 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.301+0000 7f88b8ff9640 1 -- 172.21.15.57:0/3878891526 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3636166233 0 0) 0x7f88bc1d7570 con 0x7f88bc13f650 2024-03-29T10:49:27.293 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.301+0000 7f88b8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:49:27.293 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.301+0000 7f88b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:27.293 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.301+0000 7f88b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:27.293 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.301+0000 7f88b8ff9640 10 monclient(hunting): my global_id is 4566 2024-03-29T10:49:27.293 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.301+0000 7f88b8ff9640 1 -- 172.21.15.57:0/3878891526 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88a8004eb0 con 0x7f88bc13f650 2024-03-29T10:49:27.293 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.301+0000 7f88b8ff9640 1 -- 172.21.15.57:0/3878891526 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1122014484 0 0) 0x7f88a8004eb0 con 0x7f88bc13f650 2024-03-29T10:49:27.293 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.301+0000 7f88b8ff9640 1 -- 172.21.15.57:0/3878891526 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88bc1d7570 con 0x7f88bc13f650 2024-03-29T10:49:27.293 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.301+0000 7f88b8ff9640 1 -- 172.21.15.57:0/3878891526 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f88b00018f0 con 0x7f88bc13f650 2024-03-29T10:49:27.293 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.301+0000 7f88b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:27.293 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.301+0000 7f88b8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:27.293 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.301+0000 7f88b8ff9640 10 monclient(hunting): dump: 2024-03-29T10:49:27.293 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:27.294 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:27.294 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:27.294 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:27.294 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:27.294 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:27.294 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:27.294 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:27.294 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:27.294 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:27.294 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.301+0000 7f88b8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:27.294 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.302+0000 7f88b8ff9640 1 -- 172.21.15.57:0/3878891526 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 224755186 0 0) 0x7f88bc1d7570 con 0x7f88bc13f650 2024-03-29T10:49:27.294 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.302+0000 7f88b8ff9640 1 -- 172.21.15.57:0/3878891526 >> v1:172.21.15.57:6790/0 conn(0x7f88bc1457f0 legacy=0x7f88bc1e2db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:49:27.294 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.302+0000 7f88b8ff9640 1 -- 172.21.15.57:0/3878891526 >> v1:172.21.15.179:6789/0 conn(0x7f88bc0a9d70 legacy=0x7f88bc1d7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:27.294 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.302+0000 7f88b8ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:49:27.295 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.302+0000 7f88b8ff9640 1 monclient: found mon.a 2024-03-29T10:49:27.295 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.302+0000 7f88b8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:27.295 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.302+0000 7f88b8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:49:27.295 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.302+0000 7f88b8ff9640 1 -- 172.21.15.57:0/3878891526 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88bc0b0bc0 con 0x7f88bc13f650 2024-03-29T10:49:27.295 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.302+0000 7f88b8ff9640 10 monclient: _finish_auth 0 2024-03-29T10:49:27.295 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.302+0000 7f88b8ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:49:27.295 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.302+0000 7f88b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:27.295 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.303+0000 7f88c2515640 5 monclient: authenticate success, global_id 4566 2024-03-29T10:49:27.295 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.303+0000 7f88c2515640 10 monclient: _renew_subs 2024-03-29T10:49:27.295 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.303+0000 7f88c2515640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:49:27.295 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.303+0000 7f88c2515640 1 -- 172.21.15.57:0/3878891526 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88bc0b0e10 con 0x7f88bc13f650 2024-03-29T10:49:27.296 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.303+0000 7f88b8ff9640 1 -- 172.21.15.57:0/3878891526 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88b0003610 con 0x7f88bc13f650 2024-03-29T10:49:27.296 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.303+0000 7f88b8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:27.296 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.303+0000 7f88b8ff9640 1 -- 172.21.15.57:0/3878891526 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f88b0003780 con 0x7f88bc13f650 2024-03-29T10:49:27.296 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.303+0000 7f88b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:27.296 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.303+0000 7f88b9ffb640 4 set_mon_vals no callback set 2024-03-29T10:49:27.296 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.303+0000 7f88b8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:27.296 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.303+0000 7f88b8ff9640 10 monclient: dump: 2024-03-29T10:49:27.296 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:27.296 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:27.296 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:27.296 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:27.296 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:27.296 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:27.296 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:27.297 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:27.297 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:27.297 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:27.297 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.303+0000 7f88c2515640 10 client.4566.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:27.297 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.303+0000 7f88c2515640 10 monclient: _renew_subs 2024-03-29T10:49:27.297 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.303+0000 7f88c2515640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:49:27.297 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.303+0000 7f88c2515640 1 -- 172.21.15.57:0/3878891526 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88bc13b930 con 0x7f88bc13f650 2024-03-29T10:49:27.297 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.303+0000 7f88c2515640 1 librados: init done 2024-03-29T10:49:27.297 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.305+0000 7f88c2515640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:27.297 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.305+0000 7f88c2515640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:27.297 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.305+0000 7f88c2515640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:49:27.298 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.306+0000 7f88c2515640 1 -- 172.21.15.57:0/3878891526 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88bc0b18b0 con 0x7f88bc13f650 2024-03-29T10:49:27.298 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.306+0000 7f88b8ff9640 1 -- 172.21.15.57:0/3878891526 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f88b0019750 con 0x7f88bc13f650 2024-03-29T10:49:27.298 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.307+0000 7f88b8ff9640 1 -- 172.21.15.57:0/3878891526 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f88b0083790 con 0x7f88bc13f650 2024-03-29T10:49:27.298 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.307+0000 7f88b8ff9640 10 client.4566.objecter ms_dispatch 0x7f88bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:27.298 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.307+0000 7f88b8ff9640 3 client.4566.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:27.298 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.307+0000 7f88b8ff9640 3 client.4566.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:27.298 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.307+0000 7f88b8ff9640 20 client.4566.objecter dump_active .. 0 homeless 2024-03-29T10:49:27.298 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.308+0000 7f88bbfff640 10 monclient: get_auth_request con 0x7f88a806dd20 auth_method 0 2024-03-29T10:49:27.299 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.310+0000 7f88b8ff9640 10 client.4566.objecter ms_handle_connect 0x7f88a806dd20 2024-03-29T10:49:27.299 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.313+0000 7f88b8ff9640 1 -- 172.21.15.57:0/3878891526 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f88b004f070 con 0x7f88bc13f650 2024-03-29T10:49:27.299 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.313+0000 7f88b8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:27.299 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.313+0000 7f88b8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:27.636 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.653+0000 7f88c2515640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:27.636 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.653+0000 7f88c2515640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:27.636 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.653+0000 7f88c2515640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:49:27.636 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.653+0000 7f88c2515640 1 -- 172.21.15.57:0/3878891526 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88bc1d7720 con 0x7f88bc13f650 2024-03-29T10:49:27.638 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.656+0000 7f88b8ff9640 1 -- 172.21.15.57:0/3878891526 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f88bc1d7720 con 0x7f88bc13f650 2024-03-29T10:49:27.639 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.656+0000 7f88b8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:27.639 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.656+0000 7f88b8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:27.639 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:49:27.639 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:27.643 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.661+0000 7f88c2515640 10 librados: watch_flush enter 2024-03-29T10:49:27.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.661+0000 7f88c2515640 10 librados: watch_flush exit 2024-03-29T10:49:27.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.661+0000 7f88c2515640 20 client.4566.objecter shutdown clearing up homeless session... 2024-03-29T10:49:27.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.661+0000 7f88c2515640 10 client.4566.objecter successfully canceled tick 2024-03-29T10:49:27.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.661+0000 7f88c2515640 1 -- 172.21.15.57:0/3878891526 >> v1:172.21.15.179:6816/2698295936 conn(0x7f88a806dd20 legacy=0x7f88a80701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:27.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.661+0000 7f88c2515640 10 monclient: shutdown 2024-03-29T10:49:27.644 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.661+0000 7f88c2515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:27.645 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.661+0000 7f88c2515640 1 -- 172.21.15.57:0/3878891526 >> v1:172.21.15.57:6789/0 conn(0x7f88bc13f650 legacy=0x7f88bc134d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:27.645 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.661+0000 7f88c2515640 1 -- 172.21.15.57:0/3878891526 shutdown_connections 2024-03-29T10:49:27.645 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.661+0000 7f88c2515640 1 -- 172.21.15.57:0/3878891526 >> 172.21.15.57:0/3878891526 conn(0x7f88bc066a00 msgr2=0x7f88bc0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:27.645 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.662+0000 7f88c2515640 1 -- 172.21.15.57:0/3878891526 shutdown_connections 2024-03-29T10:49:27.645 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.662+0000 7f88c2515640 1 -- 172.21.15.57:0/3878891526 wait complete. 2024-03-29T10:49:27.645 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:27.662+0000 7f88c2515640 1 librados: shutdown 2024-03-29T10:49:27.659 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:34.466 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:34.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.621+0000 7f0adc754640 10 monclient: get_monmap_and_config 2024-03-29T10:49:34.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.621+0000 7f0adc754640 10 monclient: build_initial_monmap 2024-03-29T10:49:34.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.621+0000 7f0adc754640 10 monclient: monmap: 2024-03-29T10:49:34.603 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:49:34.603 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:34.603 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:49:34.621884+0000 2024-03-29T10:49:34.603 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:49:34.621884+0000 2024-03-29T10:49:34.603 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:34.603 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:34.603 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:49:34.603 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:49:34.603 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:49:34.604 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:34.605 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.622+0000 7f0adc754640 1 Processor -- start 2024-03-29T10:49:34.605 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.622+0000 7f0adc754640 1 -- start start 2024-03-29T10:49:34.605 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.622+0000 7f0adc754640 10 monclient: init 2024-03-29T10:49:34.605 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.623+0000 7f0adc754640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:34.605 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.623+0000 7f0adc754640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:49:34.605 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.623+0000 7f0adc754640 10 monclient(hunting): picked mon.noname-c con 0x7f0ad40b0770 addr v1:172.21.15.179:6789/0 2024-03-29T10:49:34.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.623+0000 7f0adc754640 10 monclient(hunting): picked mon.noname-b con 0x7f0ad414a700 addr v1:172.21.15.57:6790/0 2024-03-29T10:49:34.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.623+0000 7f0adc754640 10 monclient(hunting): picked mon.noname-a con 0x7f0ad40ac5a0 addr v1:172.21.15.57:6789/0 2024-03-29T10:49:34.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.623+0000 7f0adc754640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad40aba70 con 0x7f0ad40ac5a0 2024-03-29T10:49:34.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.623+0000 7f0adc754640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad40aecb0 con 0x7f0ad40b0770 2024-03-29T10:49:34.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.623+0000 7f0adc754640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad40af150 con 0x7f0ad414a700 2024-03-29T10:49:34.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.623+0000 7f0adc754640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:34.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.623+0000 7f0adc754640 10 monclient(hunting): authenticate will time out at 1526.799298s 2024-03-29T10:49:34.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.623+0000 7f0ad9cef640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f0ad414a700 0x7f0ad40abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:33272/0 (socket says 172.21.15.57:33272) 2024-03-29T10:49:34.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.623+0000 7f0ad9cef640 1 -- 172.21.15.57:0/362495423 learned_addr learned my addr 172.21.15.57:0/362495423 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:49:34.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.623+0000 7f0ad94ee640 1 -- 172.21.15.57:0/362495423 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796864129 0 0) 0x7f0ad40af150 con 0x7f0ad414a700 2024-03-29T10:49:34.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.623+0000 7f0ad94ee640 10 monclient(hunting): _negotiate 2024-03-29T10:49:34.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.623+0000 7f0ad94ee640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:34.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.623+0000 7f0ad94ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:34.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.623+0000 7f0ad94ee640 10 monclient(hunting): my global_id is 4619 2024-03-29T10:49:34.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.623+0000 7f0ad94ee640 1 -- 172.21.15.57:0/362495423 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ac0003520 con 0x7f0ad414a700 2024-03-29T10:49:34.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.623+0000 7f0ad94ee640 1 -- 172.21.15.57:0/362495423 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 27724465 0 0) 0x7f0ac0003520 con 0x7f0ad414a700 2024-03-29T10:49:34.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.623+0000 7f0ad94ee640 1 -- 172.21.15.57:0/362495423 >> v1:172.21.15.179:6789/0 conn(0x7f0ad40b0770 legacy=0x7f0ad4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:49:34.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.623+0000 7f0ad94ee640 1 -- 172.21.15.57:0/362495423 >> v1:172.21.15.57:6789/0 conn(0x7f0ad40ac5a0 legacy=0x7f0ad4151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:34.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.626+0000 7f0ad94ee640 10 monclient: _finish_hunting 0 2024-03-29T10:49:34.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.626+0000 7f0ad94ee640 1 monclient: found mon.noname-b 2024-03-29T10:49:34.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.626+0000 7f0ad94ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:34.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.626+0000 7f0ad94ee640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:49:34.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.626+0000 7f0ad94ee640 1 -- 172.21.15.57:0/362495423 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad40ac9d0 con 0x7f0ad414a700 2024-03-29T10:49:34.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.626+0000 7f0ad94ee640 10 monclient: _finish_auth 0 2024-03-29T10:49:34.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.626+0000 7f0ad94ee640 10 monclient: _check_auth_tickets 2024-03-29T10:49:34.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.626+0000 7f0ad94ee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:34.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.626+0000 7f0ad94ee640 1 -- 172.21.15.57:0/362495423 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f0acc0019f0 con 0x7f0ad414a700 2024-03-29T10:49:34.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.626+0000 7f0ad94ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:34.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.626+0000 7f0ad94ee640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:34.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.626+0000 7f0ad94ee640 10 monclient: dump: 2024-03-29T10:49:34.608 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:34.608 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:34.609 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:34.609 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:34.609 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:34.609 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:34.609 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:34.609 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:34.609 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:34.609 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:34.609 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.626+0000 7f0adc754640 5 monclient: authenticate success, global_id 4619 2024-03-29T10:49:34.609 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.626+0000 7f0adc754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:34.610 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.626+0000 7f0ad94ee640 1 -- 172.21.15.57:0/362495423 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0acc001c50 con 0x7f0ad414a700 2024-03-29T10:49:34.611 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.626+0000 7f0ad94ee640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:34.611 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.626+0000 7f0ad94ee640 1 -- 172.21.15.57:0/362495423 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f0acc0020a0 con 0x7f0ad414a700 2024-03-29T10:49:34.611 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.626+0000 7f0ad94ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:34.611 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.626+0000 7f0ad94ee640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:34.611 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.626+0000 7f0ad94ee640 10 monclient: dump: 2024-03-29T10:49:34.611 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:34.612 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:34.612 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:34.612 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:34.612 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:34.612 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:34.612 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:34.612 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:34.612 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:34.612 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:34.612 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.626+0000 7f0adc754640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:34.612 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.626+0000 7f0adc754640 4 set_mon_vals no callback set 2024-03-29T10:49:34.612 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.627+0000 7f0adc754640 10 monclient: shutdown 2024-03-29T10:49:34.612 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.627+0000 7f0adc754640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:34.612 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.627+0000 7f0adc754640 1 -- 172.21.15.57:0/362495423 >> v1:172.21.15.57:6790/0 conn(0x7f0ad414a700 legacy=0x7f0ad40abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:34.612 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.628+0000 7f0adc754640 1 -- 172.21.15.57:0/362495423 shutdown_connections 2024-03-29T10:49:34.613 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.628+0000 7f0adc754640 1 -- 172.21.15.57:0/362495423 >> 172.21.15.57:0/362495423 conn(0x7f0ad406b660 msgr2=0x7f0ad406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:34.613 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.628+0000 7f0adc754640 1 -- 172.21.15.57:0/362495423 shutdown_connections 2024-03-29T10:49:34.618 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.628+0000 7f0adc754640 1 -- 172.21.15.57:0/362495423 wait complete. 2024-03-29T10:49:34.618 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.629+0000 7f0adc754640 10 monclient: build_initial_monmap 2024-03-29T10:49:34.618 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.629+0000 7f0adc754640 10 monclient: monmap: 2024-03-29T10:49:34.618 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:49:34.618 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:34.618 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:49:34.618 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:49:34.618 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:34.618 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:34.618 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:49:34.619 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:49:34.619 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:49:34.619 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:34.619 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.629+0000 7f0adc754640 1 librados: starting msgr at 2024-03-29T10:49:34.619 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.629+0000 7f0adc754640 1 librados: starting objecter 2024-03-29T10:49:34.619 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.629+0000 7f0adc754640 1 Processor -- start 2024-03-29T10:49:34.619 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.630+0000 7f0adc754640 1 -- start start 2024-03-29T10:49:34.619 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.630+0000 7f0adc754640 1 librados: setting wanted keys 2024-03-29T10:49:34.619 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.630+0000 7f0adc754640 1 librados: calling monclient init 2024-03-29T10:49:34.619 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.630+0000 7f0adc754640 10 monclient: init 2024-03-29T10:49:34.619 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.631+0000 7f0adc754640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:34.620 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.631+0000 7f0adc754640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:49:34.620 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.631+0000 7f0adc754640 10 monclient(hunting): picked mon.noname-c con 0x7f0ad40ac5a0 addr v1:172.21.15.57:6790/0 2024-03-29T10:49:34.620 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.631+0000 7f0adc754640 10 monclient(hunting): picked mon.noname-b con 0x7f0ad40b0770 addr v1:172.21.15.179:6789/0 2024-03-29T10:49:34.620 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.631+0000 7f0adc754640 10 monclient(hunting): picked mon.noname-a con 0x7f0ad414a700 addr v1:172.21.15.57:6789/0 2024-03-29T10:49:34.620 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.631+0000 7f0adc754640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad4127590 con 0x7f0ad414a700 2024-03-29T10:49:34.620 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.631+0000 7f0adc754640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad41282f0 con 0x7f0ad40b0770 2024-03-29T10:49:34.620 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.631+0000 7f0adc754640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad41285d0 con 0x7f0ad40ac5a0 2024-03-29T10:49:34.620 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.631+0000 7f0ada4f0640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f0ad40ac5a0 0x7f0ad4126170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:33702/0 (socket says 172.21.15.57:33702) 2024-03-29T10:49:34.620 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.631+0000 7f0ada4f0640 1 -- 172.21.15.57:0/2388245018 learned_addr learned my addr 172.21.15.57:0/2388245018 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:49:34.620 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.631+0000 7f0adc754640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:34.620 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.631+0000 7f0adc754640 10 monclient(hunting): authenticate will time out at 1526.807326s 2024-03-29T10:49:34.620 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.631+0000 7f0acaffd640 10 client.?.objecter ms_handle_connect 0x7f0ad40ac5a0 2024-03-29T10:49:34.620 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.631+0000 7f0acaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:34.620 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.631+0000 7f0acaffd640 1 -- 172.21.15.57:0/2388245018 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1685255985 0 0) 0x7f0ad41285d0 con 0x7f0ad40ac5a0 2024-03-29T10:49:34.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.631+0000 7f0acaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:34.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.631+0000 7f0acaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:34.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.631+0000 7f0acaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:34.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.631+0000 7f0acaffd640 10 monclient(hunting): my global_id is 4622 2024-03-29T10:49:34.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.631+0000 7f0acaffd640 1 -- 172.21.15.57:0/2388245018 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ab8004eb0 con 0x7f0ad40ac5a0 2024-03-29T10:49:34.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.631+0000 7f0acaffd640 10 client.?.objecter ms_handle_connect 0x7f0ad40b0770 2024-03-29T10:49:34.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.631+0000 7f0acaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:34.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.632+0000 7f0acaffd640 1 -- 172.21.15.57:0/2388245018 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280782357 0 0) 0x7f0ad41282f0 con 0x7f0ad40b0770 2024-03-29T10:49:34.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.632+0000 7f0acaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:34.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.632+0000 7f0acaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:34.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.632+0000 7f0acaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:34.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.632+0000 7f0acaffd640 10 monclient(hunting): my global_id is 4564 2024-03-29T10:49:34.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.632+0000 7f0acaffd640 1 -- 172.21.15.57:0/2388245018 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ad41285d0 con 0x7f0ad40b0770 2024-03-29T10:49:34.621 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.632+0000 7f0acaffd640 1 -- 172.21.15.57:0/2388245018 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3069558364 0 0) 0x7f0ab8004eb0 con 0x7f0ad40ac5a0 2024-03-29T10:49:34.622 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.632+0000 7f0acaffd640 1 -- 172.21.15.57:0/2388245018 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ad41282f0 con 0x7f0ad40ac5a0 2024-03-29T10:49:34.622 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.632+0000 7f0acaffd640 1 -- 172.21.15.57:0/2388245018 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f0ad0001960 con 0x7f0ad40ac5a0 2024-03-29T10:49:34.622 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.632+0000 7f0acaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:34.622 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.632+0000 7f0acaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:34.622 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.632+0000 7f0acaffd640 10 monclient(hunting): dump: 2024-03-29T10:49:34.622 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:34.622 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:34.622 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:34.622 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:34.622 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:34.622 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:34.622 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:34.622 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:34.622 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:34.623 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:34.623 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.632+0000 7f0acaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:34.623 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.632+0000 7f0acaffd640 1 -- 172.21.15.57:0/2388245018 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 795617075 0 0) 0x7f0ad41282f0 con 0x7f0ad40ac5a0 2024-03-29T10:49:34.623 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.632+0000 7f0acaffd640 1 -- 172.21.15.57:0/2388245018 >> v1:172.21.15.179:6789/0 conn(0x7f0ad40b0770 legacy=0x7f0ad41db950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:34.623 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.632+0000 7f0acaffd640 1 -- 172.21.15.57:0/2388245018 >> v1:172.21.15.57:6789/0 conn(0x7f0ad414a700 legacy=0x7f0ad41e6190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:49:34.623 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.635+0000 7f0acaffd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:34.623 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.635+0000 7f0acaffd640 1 monclient: found mon.c 2024-03-29T10:49:34.623 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.635+0000 7f0acaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:34.623 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.635+0000 7f0acaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:49:34.623 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.635+0000 7f0acaffd640 1 -- 172.21.15.57:0/2388245018 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad4126f90 con 0x7f0ad40ac5a0 2024-03-29T10:49:34.623 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.635+0000 7f0acaffd640 10 monclient: _finish_auth 0 2024-03-29T10:49:34.623 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.635+0000 7f0acaffd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:34.623 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.635+0000 7f0acaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:34.623 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.635+0000 7f0adc754640 5 monclient: authenticate success, global_id 4622 2024-03-29T10:49:34.623 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.635+0000 7f0adc754640 10 monclient: _renew_subs 2024-03-29T10:49:34.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.635+0000 7f0adc754640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:49:34.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.635+0000 7f0adc754640 1 -- 172.21.15.57:0/2388245018 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad4127240 con 0x7f0ad40ac5a0 2024-03-29T10:49:34.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.635+0000 7f0adc754640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:34.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.635+0000 7f0adc754640 10 monclient: _renew_subs 2024-03-29T10:49:34.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.635+0000 7f0adc754640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:49:34.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.635+0000 7f0adc754640 1 -- 172.21.15.57:0/2388245018 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad41dc240 con 0x7f0ad40ac5a0 2024-03-29T10:49:34.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.635+0000 7f0adc754640 1 librados: init done 2024-03-29T10:49:34.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.635+0000 7f0acaffd640 1 -- 172.21.15.57:0/2388245018 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ad00035f0 con 0x7f0ad40ac5a0 2024-03-29T10:49:34.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.635+0000 7f0acaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:34.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.635+0000 7f0acaffd640 1 -- 172.21.15.57:0/2388245018 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f0ad0003940 con 0x7f0ad40ac5a0 2024-03-29T10:49:34.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.635+0000 7f0acaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:34.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.635+0000 7f0acaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:34.624 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.635+0000 7f0acaffd640 10 monclient: dump: 2024-03-29T10:49:34.624 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:34.625 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:34.625 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:34.625 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:34.625 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:34.625 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:34.625 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:34.625 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:34.625 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:34.625 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:34.626 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.636+0000 7f0acbfff640 4 set_mon_vals no callback set 2024-03-29T10:49:34.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.636+0000 7f0adc754640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:34.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.637+0000 7f0adc754640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:34.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.637+0000 7f0adc754640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:49:34.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.637+0000 7f0adc754640 1 -- 172.21.15.57:0/2388245018 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0abc005080 con 0x7f0ad40ac5a0 2024-03-29T10:49:34.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.637+0000 7f0acaffd640 1 -- 172.21.15.57:0/2388245018 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f0ad0003aa0 con 0x7f0ad40ac5a0 2024-03-29T10:49:34.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.638+0000 7f0acaffd640 1 -- 172.21.15.57:0/2388245018 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1643998787 0 0) 0x7f0ad0084a30 con 0x7f0ad40ac5a0 2024-03-29T10:49:34.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.638+0000 7f0acaffd640 10 client.4622.objecter ms_dispatch 0x7f0ad4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:49:34.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.638+0000 7f0acaffd640 3 client.4622.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:49:34.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.638+0000 7f0acaffd640 3 client.4622.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:49:34.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.638+0000 7f0acaffd640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-29T10:49:34.627 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.638+0000 7f0ad9cef640 10 monclient: get_auth_request con 0x7f0ab806dd30 auth_method 0 2024-03-29T10:49:34.628 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.639+0000 7f0acaffd640 10 client.4622.objecter ms_handle_connect 0x7f0ab806dd30 2024-03-29T10:49:34.628 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.646+0000 7f0acaffd640 1 -- 172.21.15.57:0/2388245018 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0ad0050310 con 0x7f0ad40ac5a0 2024-03-29T10:49:34.628 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.646+0000 7f0acaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:34.628 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.646+0000 7f0acaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:34.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.968+0000 7f0adc754640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:34.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.968+0000 7f0adc754640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:34.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.968+0000 7f0adc754640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:49:34.950 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.968+0000 7f0adc754640 1 -- 172.21.15.57:0/2388245018 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0abc005330 con 0x7f0ad40ac5a0 2024-03-29T10:49:34.951 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:49:34.951 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:39:37.434056+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:39:37.434056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:34.951 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.968+0000 7f0acaffd640 1 -- 172.21.15.57:0/2388245018 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 4275761619) 0x7f0ad0052240 con 0x7f0ad40ac5a0 2024-03-29T10:49:34.952 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.968+0000 7f0acaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:34.952 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.968+0000 7f0acaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:34.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.972+0000 7f0adc754640 10 librados: watch_flush enter 2024-03-29T10:49:34.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.972+0000 7f0adc754640 10 librados: watch_flush exit 2024-03-29T10:49:34.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.972+0000 7f0adc754640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-29T10:49:34.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.972+0000 7f0adc754640 10 client.4622.objecter successfully canceled tick 2024-03-29T10:49:34.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.972+0000 7f0adc754640 1 -- 172.21.15.57:0/2388245018 >> v1:172.21.15.179:6816/2698295936 conn(0x7f0ab806dd30 legacy=0x7f0ab80701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:34.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.972+0000 7f0adc754640 10 monclient: shutdown 2024-03-29T10:49:34.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.972+0000 7f0adc754640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:34.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.972+0000 7f0adc754640 1 -- 172.21.15.57:0/2388245018 >> v1:172.21.15.57:6790/0 conn(0x7f0ad40ac5a0 legacy=0x7f0ad4126170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:34.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.972+0000 7f0adc754640 1 -- 172.21.15.57:0/2388245018 shutdown_connections 2024-03-29T10:49:34.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.972+0000 7f0adc754640 1 -- 172.21.15.57:0/2388245018 >> 172.21.15.57:0/2388245018 conn(0x7f0ad406b660 msgr2=0x7f0ad40abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:34.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.972+0000 7f0adc754640 1 -- 172.21.15.57:0/2388245018 shutdown_connections 2024-03-29T10:49:34.958 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.972+0000 7f0adc754640 1 -- 172.21.15.57:0/2388245018 wait complete. 2024-03-29T10:49:34.958 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:34.972+0000 7f0adc754640 1 librados: shutdown 2024-03-29T10:49:34.971 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:41.777 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:41.911 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.929+0000 7fe0b570d640 10 monclient: get_monmap_and_config 2024-03-29T10:49:41.911 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.929+0000 7fe0b570d640 10 monclient: build_initial_monmap 2024-03-29T10:49:41.911 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.929+0000 7fe0b570d640 10 monclient: monmap: 2024-03-29T10:49:41.911 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:49:41.911 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:41.911 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:49:41.930322+0000 2024-03-29T10:49:41.911 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:49:41.930322+0000 2024-03-29T10:49:41.911 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:41.911 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:41.912 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:49:41.912 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:49:41.912 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:49:41.912 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:41.912 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.930+0000 7fe0b570d640 1 Processor -- start 2024-03-29T10:49:41.912 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.930+0000 7fe0b570d640 1 -- start start 2024-03-29T10:49:41.912 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.930+0000 7fe0b570d640 10 monclient: init 2024-03-29T10:49:41.915 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.931+0000 7fe0b570d640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:41.915 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.931+0000 7fe0b570d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:49:41.915 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.931+0000 7fe0b570d640 10 monclient(hunting): picked mon.noname-c con 0x7fe0b0149c50 addr v1:172.21.15.179:6789/0 2024-03-29T10:49:41.915 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.931+0000 7fe0b570d640 10 monclient(hunting): picked mon.noname-a con 0x7fe0b00a9cc0 addr v1:172.21.15.57:6789/0 2024-03-29T10:49:41.915 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.931+0000 7fe0b570d640 10 monclient(hunting): picked mon.noname-b con 0x7fe0b0143d00 addr v1:172.21.15.57:6790/0 2024-03-29T10:49:41.915 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.931+0000 7fe0b570d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b00ab9c0 con 0x7fe0b00a9cc0 2024-03-29T10:49:41.915 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.931+0000 7fe0b570d640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b00abc10 con 0x7fe0b0149c50 2024-03-29T10:49:41.915 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.931+0000 7fe0b570d640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b00aad20 con 0x7fe0b0143d00 2024-03-29T10:49:41.915 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.931+0000 7fe0b570d640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:41.915 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.931+0000 7fe0b570d640 10 monclient(hunting): authenticate will time out at 1534.107873s 2024-03-29T10:49:41.915 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.931+0000 7fe0af7fe640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7fe0b0143d00 0x7fe0b0151d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:50532/0 (socket says 172.21.15.57:50532) 2024-03-29T10:49:41.915 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.931+0000 7fe0af7fe640 1 -- 172.21.15.57:0/501723913 learned_addr learned my addr 172.21.15.57:0/501723913 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:49:41.916 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.932+0000 7fe0adffb640 1 -- 172.21.15.57:0/501723913 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123777340 0 0) 0x7fe0b00aad20 con 0x7fe0b0143d00 2024-03-29T10:49:41.916 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.932+0000 7fe0adffb640 10 monclient(hunting): _negotiate 2024-03-29T10:49:41.916 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.932+0000 7fe0adffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:41.916 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.932+0000 7fe0adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:41.916 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.932+0000 7fe0adffb640 10 monclient(hunting): my global_id is 4625 2024-03-29T10:49:41.916 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.932+0000 7fe0adffb640 1 -- 172.21.15.57:0/501723913 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe098003520 con 0x7fe0b0143d00 2024-03-29T10:49:41.916 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.932+0000 7fe0adffb640 1 -- 172.21.15.57:0/501723913 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 89642224 0 0) 0x7fe098003520 con 0x7fe0b0143d00 2024-03-29T10:49:41.916 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.932+0000 7fe0adffb640 1 -- 172.21.15.57:0/501723913 >> v1:172.21.15.179:6789/0 conn(0x7fe0b0149c50 legacy=0x7fe0b00ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:41.916 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.932+0000 7fe0adffb640 1 -- 172.21.15.57:0/501723913 >> v1:172.21.15.57:6789/0 conn(0x7fe0b00a9cc0 legacy=0x7fe0b01435f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:41.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.934+0000 7fe0adffb640 10 monclient: _finish_hunting 0 2024-03-29T10:49:41.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.934+0000 7fe0adffb640 1 monclient: found mon.noname-b 2024-03-29T10:49:41.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.934+0000 7fe0adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:41.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.934+0000 7fe0adffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:49:41.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.934+0000 7fe0adffb640 1 -- 172.21.15.57:0/501723913 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b00aa980 con 0x7fe0b0143d00 2024-03-29T10:49:41.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.934+0000 7fe0adffb640 10 monclient: _finish_auth 0 2024-03-29T10:49:41.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.934+0000 7fe0adffb640 10 monclient: _check_auth_tickets 2024-03-29T10:49:41.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.934+0000 7fe0adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:41.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.934+0000 7fe0b570d640 5 monclient: authenticate success, global_id 4625 2024-03-29T10:49:41.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.934+0000 7fe0adffb640 1 -- 172.21.15.57:0/501723913 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fe0a8001a10 con 0x7fe0b0143d00 2024-03-29T10:49:41.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.934+0000 7fe0b570d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:41.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.934+0000 7fe0adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:41.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.934+0000 7fe0adffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:41.919 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.934+0000 7fe0adffb640 10 monclient: dump: 2024-03-29T10:49:41.920 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:41.920 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:41.920 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:41.920 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:41.920 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:41.920 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:41.920 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:41.920 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:41.920 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:41.920 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:41.921 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.934+0000 7fe0b570d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:41.921 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.935+0000 7fe0adffb640 1 -- 172.21.15.57:0/501723913 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0a8001ca0 con 0x7fe0b0143d00 2024-03-29T10:49:41.921 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.935+0000 7fe0adffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:41.921 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.935+0000 7fe0b570d640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:41.921 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.935+0000 7fe0b570d640 4 set_mon_vals no callback set 2024-03-29T10:49:41.921 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.935+0000 7fe0adffb640 1 -- 172.21.15.57:0/501723913 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fe0a80020b0 con 0x7fe0b0143d00 2024-03-29T10:49:41.921 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.936+0000 7fe0adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:41.921 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.936+0000 7fe0adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:41.921 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.936+0000 7fe0adffb640 10 monclient: dump: 2024-03-29T10:49:41.922 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:41.922 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:41.922 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:41.922 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:41.922 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:41.922 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:41.922 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:41.922 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:41.922 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:41.922 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:41.922 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.936+0000 7fe0b570d640 10 monclient: shutdown 2024-03-29T10:49:41.922 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.936+0000 7fe0b570d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:41.922 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.936+0000 7fe0b570d640 1 -- 172.21.15.57:0/501723913 >> v1:172.21.15.57:6790/0 conn(0x7fe0b0143d00 legacy=0x7fe0b0151d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:41.922 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.936+0000 7fe0b570d640 1 -- 172.21.15.57:0/501723913 shutdown_connections 2024-03-29T10:49:41.923 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.936+0000 7fe0b570d640 1 -- 172.21.15.57:0/501723913 >> 172.21.15.57:0/501723913 conn(0x7fe0b00b3c50 msgr2=0x7fe0b00b4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:41.923 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.936+0000 7fe0b570d640 1 -- 172.21.15.57:0/501723913 shutdown_connections 2024-03-29T10:49:41.923 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.936+0000 7fe0b570d640 1 -- 172.21.15.57:0/501723913 wait complete. 2024-03-29T10:49:41.923 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.937+0000 7fe0b570d640 10 monclient: build_initial_monmap 2024-03-29T10:49:41.923 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.937+0000 7fe0b570d640 10 monclient: monmap: 2024-03-29T10:49:41.923 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:49:41.923 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:41.923 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:49:41.923 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:49:41.923 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:41.923 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:41.923 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:49:41.923 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:49:41.923 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:49:41.923 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:41.927 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.937+0000 7fe0b570d640 1 librados: starting msgr at 2024-03-29T10:49:41.927 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.937+0000 7fe0b570d640 1 librados: starting objecter 2024-03-29T10:49:41.927 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.937+0000 7fe0b570d640 1 Processor -- start 2024-03-29T10:49:41.927 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.937+0000 7fe0b570d640 1 -- start start 2024-03-29T10:49:41.927 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.937+0000 7fe0b570d640 1 librados: setting wanted keys 2024-03-29T10:49:41.928 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.937+0000 7fe0b570d640 1 librados: calling monclient init 2024-03-29T10:49:41.928 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.937+0000 7fe0b570d640 10 monclient: init 2024-03-29T10:49:41.928 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.938+0000 7fe0b570d640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:41.928 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.938+0000 7fe0b570d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:49:41.928 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.938+0000 7fe0b570d640 10 monclient(hunting): picked mon.noname-c con 0x7fe0b00a9cc0 addr v1:172.21.15.57:6790/0 2024-03-29T10:49:41.928 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.938+0000 7fe0b570d640 10 monclient(hunting): picked mon.noname-a con 0x7fe0b0143d00 addr v1:172.21.15.57:6789/0 2024-03-29T10:49:41.928 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.939+0000 7fe0aeffd640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7fe0b00a9cc0 0x7fe0b0146ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:51140/0 (socket says 172.21.15.57:51140) 2024-03-29T10:49:41.928 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0b570d640 10 monclient(hunting): picked mon.noname-b con 0x7fe0b0149c50 addr v1:172.21.15.179:6789/0 2024-03-29T10:49:41.928 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0b570d640 1 -- 172.21.15.57:0/1318323500 --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b01db490 con 0x7fe0b0143d00 2024-03-29T10:49:41.928 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0aeffd640 1 -- 172.21.15.57:0/1318323500 learned_addr learned my addr 172.21.15.57:0/1318323500 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:49:41.928 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0b570d640 1 -- 172.21.15.57:0/1318323500 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b006fad0 con 0x7fe0b0149c50 2024-03-29T10:49:41.928 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0b570d640 1 -- 172.21.15.57:0/1318323500 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b0144260 con 0x7fe0b00a9cc0 2024-03-29T10:49:41.928 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0b570d640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:41.928 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0b570d640 10 monclient(hunting): authenticate will time out at 1534.116186s 2024-03-29T10:49:41.929 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0977fe640 10 client.?.objecter ms_handle_connect 0x7fe0b00a9cc0 2024-03-29T10:49:41.929 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0977fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:41.929 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0977fe640 10 client.?.objecter ms_handle_connect 0x7fe0b0149c50 2024-03-29T10:49:41.929 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0977fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:41.929 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0977fe640 1 -- 172.21.15.57:0/1318323500 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715626780 0 0) 0x7fe0b0144260 con 0x7fe0b00a9cc0 2024-03-29T10:49:41.929 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0977fe640 10 monclient(hunting): _negotiate 2024-03-29T10:49:41.929 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:41.929 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:41.929 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0977fe640 10 monclient(hunting): my global_id is 4628 2024-03-29T10:49:41.929 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0977fe640 1 -- 172.21.15.57:0/1318323500 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe09c004eb0 con 0x7fe0b00a9cc0 2024-03-29T10:49:41.929 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0977fe640 1 -- 172.21.15.57:0/1318323500 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 75702939 0 0) 0x7fe09c004eb0 con 0x7fe0b00a9cc0 2024-03-29T10:49:41.929 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0977fe640 1 -- 172.21.15.57:0/1318323500 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0b0144260 con 0x7fe0b00a9cc0 2024-03-29T10:49:41.929 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0977fe640 1 -- 172.21.15.57:0/1318323500 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fe0a4001930 con 0x7fe0b00a9cc0 2024-03-29T10:49:41.929 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:41.930 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0977fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:41.930 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0977fe640 10 monclient(hunting): dump: 2024-03-29T10:49:41.930 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:41.930 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:41.930 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:41.930 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:41.930 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:41.930 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:41.930 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:41.930 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:41.930 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:41.930 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:41.930 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0977fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:41.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0977fe640 1 -- 172.21.15.57:0/1318323500 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1805566629 0 0) 0x7fe0b0144260 con 0x7fe0b00a9cc0 2024-03-29T10:49:41.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.940+0000 7fe0977fe640 1 -- 172.21.15.57:0/1318323500 >> v1:172.21.15.179:6789/0 conn(0x7fe0b0149c50 legacy=0x7fe0b01db920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:41.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0977fe640 1 -- 172.21.15.57:0/1318323500 >> v1:172.21.15.57:6789/0 conn(0x7fe0b0143d00 legacy=0x7fe0b0072360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:49:41.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0977fe640 10 monclient: _finish_hunting 0 2024-03-29T10:49:41.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0977fe640 1 monclient: found mon.c 2024-03-29T10:49:41.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:41.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0977fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:49:41.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0977fe640 1 -- 172.21.15.57:0/1318323500 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b006e900 con 0x7fe0b00a9cc0 2024-03-29T10:49:41.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0977fe640 10 monclient: _finish_auth 0 2024-03-29T10:49:41.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0977fe640 10 monclient: _check_auth_tickets 2024-03-29T10:49:41.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:41.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0b570d640 5 monclient: authenticate success, global_id 4628 2024-03-29T10:49:41.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0b570d640 10 monclient: _renew_subs 2024-03-29T10:49:41.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0b570d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:49:41.931 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0b570d640 1 -- 172.21.15.57:0/1318323500 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0b0072a70 con 0x7fe0b00a9cc0 2024-03-29T10:49:41.932 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0b570d640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:41.932 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0b570d640 10 monclient: _renew_subs 2024-03-29T10:49:41.932 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0b570d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:49:41.932 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0b570d640 1 -- 172.21.15.57:0/1318323500 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0b006e4e0 con 0x7fe0b00a9cc0 2024-03-29T10:49:41.932 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0b570d640 1 librados: init done 2024-03-29T10:49:41.932 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0977fe640 1 -- 172.21.15.57:0/1318323500 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0a4003610 con 0x7fe0b00a9cc0 2024-03-29T10:49:41.932 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:41.932 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0977fe640 1 -- 172.21.15.57:0/1318323500 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fe0a40039e0 con 0x7fe0b00a9cc0 2024-03-29T10:49:41.932 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:41.932 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:41.932 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0ad7fa640 4 set_mon_vals no callback set 2024-03-29T10:49:41.932 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.941+0000 7fe0977fe640 10 monclient: dump: 2024-03-29T10:49:41.932 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:41.932 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:41.932 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:41.933 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:41.933 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:41.933 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:41.933 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:41.933 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:41.933 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:41.933 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:41.933 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.943+0000 7fe0b570d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:41.933 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.943+0000 7fe0977fe640 1 -- 172.21.15.57:0/1318323500 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fe0a4003b40 con 0x7fe0b00a9cc0 2024-03-29T10:49:41.933 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.943+0000 7fe0b570d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:41.933 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.943+0000 7fe0b570d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:49:41.933 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.943+0000 7fe0b570d640 1 -- 172.21.15.57:0/1318323500 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe090005080 con 0x7fe0b00a9cc0 2024-03-29T10:49:41.933 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.944+0000 7fe0977fe640 1 -- 172.21.15.57:0/1318323500 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3486612304 0 0) 0x7fe0a4084a10 con 0x7fe0b00a9cc0 2024-03-29T10:49:41.933 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.944+0000 7fe0977fe640 10 client.4628.objecter ms_dispatch 0x7fe0b0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:49:41.933 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.944+0000 7fe0977fe640 3 client.4628.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:49:41.934 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.944+0000 7fe0977fe640 3 client.4628.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:49:41.934 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.944+0000 7fe0977fe640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-29T10:49:41.934 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.944+0000 7fe0ae7fc640 10 monclient: get_auth_request con 0x7fe09c06dc70 auth_method 0 2024-03-29T10:49:41.934 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.944+0000 7fe0977fe640 10 client.4628.objecter ms_handle_connect 0x7fe09c06dc70 2024-03-29T10:49:41.934 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.951+0000 7fe0977fe640 1 -- 172.21.15.57:0/1318323500 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe0a40502f0 con 0x7fe0b00a9cc0 2024-03-29T10:49:41.934 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.951+0000 7fe0977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:41.934 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:41.951+0000 7fe0977fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:42.247 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:42.265+0000 7fe0b570d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:42.247 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:42.265+0000 7fe0b570d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:42.247 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:42.265+0000 7fe0b570d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:49:42.247 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:42.265+0000 7fe0b570d640 1 -- 172.21.15.57:0/1318323500 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe090005330 con 0x7fe0b00a9cc0 2024-03-29T10:49:42.247 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:42.266+0000 7fe0977fe640 1 -- 172.21.15.57:0/1318323500 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2357246973) 0x7fe0a4052220 con 0x7fe0b00a9cc0 2024-03-29T10:49:42.247 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:42.266+0000 7fe0977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:42.248 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:42.266+0000 7fe0977fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:42.248 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:49:42.248 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":11,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:49:40.047000+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:49:40.047000+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:42.250 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:42.269+0000 7fe0b570d640 10 librados: watch_flush enter 2024-03-29T10:49:42.250 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:42.269+0000 7fe0b570d640 10 librados: watch_flush exit 2024-03-29T10:49:42.251 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:42.269+0000 7fe0b570d640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-29T10:49:42.251 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:42.269+0000 7fe0b570d640 10 client.4628.objecter successfully canceled tick 2024-03-29T10:49:42.251 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:42.269+0000 7fe0b570d640 1 -- 172.21.15.57:0/1318323500 >> v1:172.21.15.179:6816/2698295936 conn(0x7fe09c06dc70 legacy=0x7fe09c070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:42.251 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:42.269+0000 7fe0b570d640 10 monclient: shutdown 2024-03-29T10:49:42.251 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:42.269+0000 7fe0b570d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:42.251 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:42.269+0000 7fe0b570d640 1 -- 172.21.15.57:0/1318323500 >> v1:172.21.15.57:6790/0 conn(0x7fe0b00a9cc0 legacy=0x7fe0b0146ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:42.252 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:42.269+0000 7fe0b570d640 1 -- 172.21.15.57:0/1318323500 shutdown_connections 2024-03-29T10:49:42.252 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:42.269+0000 7fe0b570d640 1 -- 172.21.15.57:0/1318323500 >> 172.21.15.57:0/1318323500 conn(0x7fe0b00b3c50 msgr2=0x7fe0b00b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:42.252 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:42.269+0000 7fe0b570d640 1 -- 172.21.15.57:0/1318323500 shutdown_connections 2024-03-29T10:49:42.252 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:42.269+0000 7fe0b570d640 1 -- 172.21.15.57:0/1318323500 wait complete. 2024-03-29T10:49:42.252 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:42.269+0000 7fe0b570d640 1 librados: shutdown 2024-03-29T10:49:42.265 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:49.072 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:49.223 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.241+0000 7feb37505640 10 monclient: get_monmap_and_config 2024-03-29T10:49:49.223 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.241+0000 7feb37505640 10 monclient: build_initial_monmap 2024-03-29T10:49:49.223 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.241+0000 7feb37505640 10 monclient: monmap: 2024-03-29T10:49:49.223 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:49:49.223 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:49.223 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:49:49.242298+0000 2024-03-29T10:49:49.223 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:49:49.242298+0000 2024-03-29T10:49:49.224 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:49.224 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:49.224 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:49:49.224 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:49:49.224 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:49:49.224 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:49.224 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.242+0000 7feb37505640 1 Processor -- start 2024-03-29T10:49:49.225 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.242+0000 7feb37505640 1 -- start start 2024-03-29T10:49:49.225 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.242+0000 7feb37505640 10 monclient: init 2024-03-29T10:49:49.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.243+0000 7feb37505640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:49.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.243+0000 7feb37505640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:49:49.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.243+0000 7feb37505640 10 monclient(hunting): picked mon.noname-a con 0x7feb28037230 addr v1:172.21.15.57:6789/0 2024-03-29T10:49:49.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.243+0000 7feb37505640 10 monclient(hunting): picked mon.noname-c con 0x7feb280341c0 addr v1:172.21.15.179:6789/0 2024-03-29T10:49:49.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.243+0000 7feb37505640 10 monclient(hunting): picked mon.noname-b con 0x7feb2802ccd0 addr v1:172.21.15.57:6790/0 2024-03-29T10:49:49.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.243+0000 7feb37505640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2802c840 con 0x7feb28037230 2024-03-29T10:49:49.227 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.243+0000 7feb37505640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2802e360 con 0x7feb280341c0 2024-03-29T10:49:49.228 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.243+0000 7feb37505640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb280c9270 con 0x7feb2802ccd0 2024-03-29T10:49:49.228 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.243+0000 7feb37505640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:49.228 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.243+0000 7feb37505640 10 monclient(hunting): authenticate will time out at 1541.419847s 2024-03-29T10:49:49.228 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.244+0000 7feb36d04640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7feb2802ccd0 0x7feb280d0f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:47212/0 (socket says 172.21.15.57:47212) 2024-03-29T10:49:49.228 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.244+0000 7feb36d04640 1 -- 172.21.15.57:0/1597924038 learned_addr learned my addr 172.21.15.57:0/1597924038 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:49:49.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.246+0000 7feb35501640 1 -- 172.21.15.57:0/1597924038 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972449934 0 0) 0x7feb280c9270 con 0x7feb2802ccd0 2024-03-29T10:49:49.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.246+0000 7feb35501640 10 monclient(hunting): _negotiate 2024-03-29T10:49:49.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.246+0000 7feb35501640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:49.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.246+0000 7feb35501640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:49.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.246+0000 7feb35501640 10 monclient(hunting): my global_id is 4631 2024-03-29T10:49:49.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.246+0000 7feb35501640 1 -- 172.21.15.57:0/1597924038 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb1c003520 con 0x7feb2802ccd0 2024-03-29T10:49:49.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.247+0000 7feb35501640 1 -- 172.21.15.57:0/1597924038 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750897101 0 0) 0x7feb2802e360 con 0x7feb280341c0 2024-03-29T10:49:49.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.247+0000 7feb35501640 10 monclient(hunting): _negotiate 2024-03-29T10:49:49.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.247+0000 7feb35501640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:49.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.247+0000 7feb35501640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:49.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.247+0000 7feb35501640 10 monclient(hunting): my global_id is 4567 2024-03-29T10:49:49.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.247+0000 7feb35501640 1 -- 172.21.15.57:0/1597924038 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb280c9270 con 0x7feb280341c0 2024-03-29T10:49:49.231 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.247+0000 7feb35501640 1 -- 172.21.15.57:0/1597924038 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 352150770 0 0) 0x7feb1c003520 con 0x7feb2802ccd0 2024-03-29T10:49:49.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.247+0000 7feb35501640 1 -- 172.21.15.57:0/1597924038 >> v1:172.21.15.179:6789/0 conn(0x7feb280341c0 legacy=0x7feb280c2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:49.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.247+0000 7feb35501640 1 -- 172.21.15.57:0/1597924038 >> v1:172.21.15.57:6789/0 conn(0x7feb28037230 legacy=0x7feb280c8b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:49:49.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.248+0000 7feb35501640 10 monclient: _finish_hunting 0 2024-03-29T10:49:49.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.248+0000 7feb35501640 1 monclient: found mon.noname-b 2024-03-29T10:49:49.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.248+0000 7feb35501640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:49.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.248+0000 7feb35501640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:49:49.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.248+0000 7feb35501640 1 -- 172.21.15.57:0/1597924038 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2802d200 con 0x7feb2802ccd0 2024-03-29T10:49:49.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.248+0000 7feb35501640 10 monclient: _finish_auth 0 2024-03-29T10:49:49.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.248+0000 7feb35501640 10 monclient: _check_auth_tickets 2024-03-29T10:49:49.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.248+0000 7feb35501640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:49.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.248+0000 7feb37505640 5 monclient: authenticate success, global_id 4631 2024-03-29T10:49:49.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.248+0000 7feb37505640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:49.232 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.249+0000 7feb35501640 1 -- 172.21.15.57:0/1597924038 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7feb200016e0 con 0x7feb2802ccd0 2024-03-29T10:49:49.234 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.249+0000 7feb35501640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:49.235 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.249+0000 7feb35501640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:49.235 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.249+0000 7feb35501640 10 monclient: dump: 2024-03-29T10:49:49.235 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:49.235 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:49.235 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:49.235 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:49.235 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:49.235 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:49.235 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:49.235 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:49.235 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:49.235 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:49.235 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.249+0000 7feb37505640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:49.236 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.249+0000 7feb35501640 1 -- 172.21.15.57:0/1597924038 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb20001970 con 0x7feb2802ccd0 2024-03-29T10:49:49.236 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.249+0000 7feb35501640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:49.236 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.249+0000 7feb37505640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:49.236 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.249+0000 7feb37505640 4 set_mon_vals no callback set 2024-03-29T10:49:49.236 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.250+0000 7feb37505640 10 monclient: shutdown 2024-03-29T10:49:49.236 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.250+0000 7feb37505640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:49.236 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.250+0000 7feb37505640 1 -- 172.21.15.57:0/1597924038 >> v1:172.21.15.57:6790/0 conn(0x7feb2802ccd0 legacy=0x7feb280d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:49.236 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.250+0000 7feb37505640 1 -- 172.21.15.57:0/1597924038 shutdown_connections 2024-03-29T10:49:49.236 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.250+0000 7feb37505640 1 -- 172.21.15.57:0/1597924038 >> 172.21.15.57:0/1597924038 conn(0x7feb2803a590 msgr2=0x7feb280367e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:49.236 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.251+0000 7feb37505640 1 -- 172.21.15.57:0/1597924038 shutdown_connections 2024-03-29T10:49:49.236 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.251+0000 7feb37505640 1 -- 172.21.15.57:0/1597924038 wait complete. 2024-03-29T10:49:49.236 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.252+0000 7feb37505640 10 monclient: build_initial_monmap 2024-03-29T10:49:49.236 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.252+0000 7feb37505640 10 monclient: monmap: 2024-03-29T10:49:49.236 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:49:49.237 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:49.237 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:49:49.237 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:49:49.237 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:49.237 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:49.237 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:49:49.237 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:49:49.237 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:49:49.237 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:49.237 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.252+0000 7feb37505640 1 librados: starting msgr at 2024-03-29T10:49:49.237 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.252+0000 7feb37505640 1 librados: starting objecter 2024-03-29T10:49:49.237 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.252+0000 7feb37505640 1 Processor -- start 2024-03-29T10:49:49.238 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.253+0000 7feb37505640 1 -- start start 2024-03-29T10:49:49.238 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.253+0000 7feb37505640 1 librados: setting wanted keys 2024-03-29T10:49:49.238 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.253+0000 7feb37505640 1 librados: calling monclient init 2024-03-29T10:49:49.238 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.253+0000 7feb37505640 10 monclient: init 2024-03-29T10:49:49.238 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.253+0000 7feb37505640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:49.238 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.253+0000 7feb37505640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:49:49.238 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.254+0000 7feb37505640 10 monclient(hunting): picked mon.noname-b con 0x7feb2802ccd0 addr v1:172.21.15.179:6789/0 2024-03-29T10:49:49.238 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.254+0000 7feb37505640 10 monclient(hunting): picked mon.noname-c con 0x7feb28037230 addr v1:172.21.15.57:6790/0 2024-03-29T10:49:49.239 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.254+0000 7feb37505640 10 monclient(hunting): picked mon.noname-a con 0x7feb280c5a20 addr v1:172.21.15.57:6789/0 2024-03-29T10:49:49.239 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.254+0000 7feb37505640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb28160a10 con 0x7feb280c5a20 2024-03-29T10:49:49.239 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.254+0000 7feb37505640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb280d4d30 con 0x7feb2802ccd0 2024-03-29T10:49:49.239 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.254+0000 7feb37505640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb280c48a0 con 0x7feb28037230 2024-03-29T10:49:49.239 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.254+0000 7feb37505640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:49.239 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.254+0000 7feb37505640 10 monclient(hunting): authenticate will time out at 1541.430117s 2024-03-29T10:49:49.239 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.256+0000 7feb35d02640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7feb28037230 0x7feb28160ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:53272/0 (socket says 172.21.15.57:53272) 2024-03-29T10:49:49.239 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.256+0000 7feb35d02640 1 -- 172.21.15.57:0/4290036703 learned_addr learned my addr 172.21.15.57:0/4290036703 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:49:49.239 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.257+0000 7feb26ffd640 10 client.?.objecter ms_handle_connect 0x7feb28037230 2024-03-29T10:49:49.239 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.257+0000 7feb26ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:49.240 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.257+0000 7feb26ffd640 1 -- 172.21.15.57:0/4290036703 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013956073 0 0) 0x7feb280c48a0 con 0x7feb28037230 2024-03-29T10:49:49.240 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.257+0000 7feb26ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:49.240 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.257+0000 7feb26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:49.240 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.257+0000 7feb26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:49.241 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.257+0000 7feb26ffd640 10 monclient(hunting): my global_id is 4634 2024-03-29T10:49:49.241 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.257+0000 7feb26ffd640 1 -- 172.21.15.57:0/4290036703 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb14004eb0 con 0x7feb28037230 2024-03-29T10:49:49.241 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.257+0000 7feb26ffd640 10 client.?.objecter ms_handle_connect 0x7feb2802ccd0 2024-03-29T10:49:49.241 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.257+0000 7feb26ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:49.241 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.258+0000 7feb26ffd640 1 -- 172.21.15.57:0/4290036703 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372952434 0 0) 0x7feb280d4d30 con 0x7feb2802ccd0 2024-03-29T10:49:49.241 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.258+0000 7feb26ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:49:49.241 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.258+0000 7feb26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:49.241 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.258+0000 7feb26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:49.241 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.258+0000 7feb26ffd640 10 monclient(hunting): my global_id is 4570 2024-03-29T10:49:49.241 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.258+0000 7feb26ffd640 1 -- 172.21.15.57:0/4290036703 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb280c48a0 con 0x7feb2802ccd0 2024-03-29T10:49:49.241 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.258+0000 7feb26ffd640 1 -- 172.21.15.57:0/4290036703 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2955265146 0 0) 0x7feb14004eb0 con 0x7feb28037230 2024-03-29T10:49:49.241 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.258+0000 7feb26ffd640 1 -- 172.21.15.57:0/4290036703 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb280d4d30 con 0x7feb28037230 2024-03-29T10:49:49.241 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.258+0000 7feb26ffd640 1 -- 172.21.15.57:0/4290036703 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7feb3004fdd0 con 0x7feb28037230 2024-03-29T10:49:49.241 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.258+0000 7feb26ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:49.242 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.258+0000 7feb26ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:49:49.242 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.258+0000 7feb26ffd640 10 monclient(hunting): dump: 2024-03-29T10:49:49.242 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:49.242 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:49.242 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:49.242 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:49.242 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:49.242 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:49.242 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:49.242 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:49.242 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:49.242 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:49.242 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.258+0000 7feb26ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:49.242 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.258+0000 7feb26ffd640 1 -- 172.21.15.57:0/4290036703 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 910969662 0 0) 0x7feb280d4d30 con 0x7feb28037230 2024-03-29T10:49:49.243 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.258+0000 7feb26ffd640 1 -- 172.21.15.57:0/4290036703 >> v1:172.21.15.179:6789/0 conn(0x7feb2802ccd0 legacy=0x7feb280c5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:49.243 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.259+0000 7feb26ffd640 1 -- 172.21.15.57:0/4290036703 >> v1:172.21.15.57:6789/0 conn(0x7feb280c5a20 legacy=0x7feb280be020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:49:49.245 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.260+0000 7feb26ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:49:49.245 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.260+0000 7feb26ffd640 1 monclient: found mon.c 2024-03-29T10:49:49.246 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.261+0000 7feb26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:49.246 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.261+0000 7feb26ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:49:49.246 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.261+0000 7feb26ffd640 1 -- 172.21.15.57:0/4290036703 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb280d24a0 con 0x7feb28037230 2024-03-29T10:49:49.246 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.261+0000 7feb26ffd640 10 monclient: _finish_auth 0 2024-03-29T10:49:49.246 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.261+0000 7feb26ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:49:49.246 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.261+0000 7feb26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:49.246 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.261+0000 7feb26ffd640 1 -- 172.21.15.57:0/4290036703 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb3004f400 con 0x7feb28037230 2024-03-29T10:49:49.246 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.261+0000 7feb37505640 5 monclient: authenticate success, global_id 4634 2024-03-29T10:49:49.246 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.261+0000 7feb37505640 10 monclient: _renew_subs 2024-03-29T10:49:49.246 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.261+0000 7feb37505640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:49:49.246 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.261+0000 7feb37505640 1 -- 172.21.15.57:0/4290036703 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb281615b0 con 0x7feb28037230 2024-03-29T10:49:49.247 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.261+0000 7feb37505640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:49.247 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.261+0000 7feb37505640 10 monclient: _renew_subs 2024-03-29T10:49:49.247 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.261+0000 7feb37505640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:49:49.247 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.261+0000 7feb37505640 1 -- 172.21.15.57:0/4290036703 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb280be730 con 0x7feb28037230 2024-03-29T10:49:49.247 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.261+0000 7feb37505640 1 librados: init done 2024-03-29T10:49:49.247 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.263+0000 7feb26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:49.247 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.263+0000 7feb26ffd640 1 -- 172.21.15.57:0/4290036703 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7feb30075710 con 0x7feb28037230 2024-03-29T10:49:49.247 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.263+0000 7feb26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:49.247 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.263+0000 7feb26ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:49:49.247 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.263+0000 7feb26ffd640 10 monclient: dump: 2024-03-29T10:49:49.247 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:49.247 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:49.247 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:49.247 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:49.247 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:49.248 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:49.248 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:49.248 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:49.248 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:49.248 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:49.248 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.263+0000 7feb24ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:49.248 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.263+0000 7feb24ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:49.248 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.263+0000 7feb24ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:49:49.248 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.263+0000 7feb34d00640 4 set_mon_vals no callback set 2024-03-29T10:49:49.248 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.264+0000 7feb24ff9640 1 -- 172.21.15.57:0/4290036703 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb28005120 con 0x7feb28037230 2024-03-29T10:49:49.248 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.267+0000 7feb26ffd640 1 -- 172.21.15.57:0/4290036703 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7feb30067bf0 con 0x7feb28037230 2024-03-29T10:49:49.250 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.268+0000 7feb26ffd640 1 -- 172.21.15.57:0/4290036703 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3486612304 0 0) 0x7feb300ff670 con 0x7feb28037230 2024-03-29T10:49:49.250 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.268+0000 7feb26ffd640 10 client.4634.objecter ms_dispatch 0x7feb30000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:49:49.250 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.268+0000 7feb26ffd640 3 client.4634.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:49:49.250 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.268+0000 7feb26ffd640 3 client.4634.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:49:49.250 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.268+0000 7feb26ffd640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-29T10:49:49.250 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.268+0000 7feb36503640 10 monclient: get_auth_request con 0x7feb1406db10 auth_method 0 2024-03-29T10:49:49.251 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.269+0000 7feb26ffd640 10 client.4634.objecter ms_handle_connect 0x7feb1406db10 2024-03-29T10:49:49.254 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.272+0000 7feb26ffd640 1 -- 172.21.15.57:0/4290036703 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feb300caf50 con 0x7feb28037230 2024-03-29T10:49:49.254 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.272+0000 7feb26ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:49.254 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.272+0000 7feb26ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:49.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.608+0000 7feb24ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:49.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.608+0000 7feb24ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:49.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.608+0000 7feb24ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:49:49.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.608+0000 7feb24ff9640 1 -- 172.21.15.57:0/4290036703 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb28160bc0 con 0x7feb28037230 2024-03-29T10:49:49.591 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.609+0000 7feb26ffd640 1 -- 172.21.15.57:0/4290036703 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2357246973) 0x7feb28160bc0 con 0x7feb28037230 2024-03-29T10:49:49.591 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.609+0000 7feb26ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:49.591 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.609+0000 7feb26ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:49.591 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:49:49.591 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":11,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:49:40.047000+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:49:40.047000+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:49.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.613+0000 7feb24ff9640 10 librados: watch_flush enter 2024-03-29T10:49:49.594 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.613+0000 7feb24ff9640 10 librados: watch_flush exit 2024-03-29T10:49:49.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.613+0000 7feb24ff9640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-29T10:49:49.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.613+0000 7feb24ff9640 10 client.4634.objecter successfully canceled tick 2024-03-29T10:49:49.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.613+0000 7feb24ff9640 1 -- 172.21.15.57:0/4290036703 >> v1:172.21.15.179:6816/2698295936 conn(0x7feb1406db10 legacy=0x7feb1406ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:49.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.613+0000 7feb24ff9640 10 monclient: shutdown 2024-03-29T10:49:49.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.613+0000 7feb24ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:49.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.613+0000 7feb24ff9640 1 -- 172.21.15.57:0/4290036703 >> v1:172.21.15.57:6790/0 conn(0x7feb28037230 legacy=0x7feb28160ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:49.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.613+0000 7feb24ff9640 1 -- 172.21.15.57:0/4290036703 shutdown_connections 2024-03-29T10:49:49.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.613+0000 7feb24ff9640 1 -- 172.21.15.57:0/4290036703 >> 172.21.15.57:0/4290036703 conn(0x7feb2803a590 msgr2=0x7feb280c2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:49.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.614+0000 7feb24ff9640 1 -- 172.21.15.57:0/4290036703 shutdown_connections 2024-03-29T10:49:49.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.614+0000 7feb24ff9640 1 -- 172.21.15.57:0/4290036703 wait complete. 2024-03-29T10:49:49.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:49.614+0000 7feb24ff9640 1 librados: shutdown 2024-03-29T10:49:49.616 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:49:56.423 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:49:56.577 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.595+0000 7fb064aac640 10 monclient: get_monmap_and_config 2024-03-29T10:49:56.577 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.595+0000 7fb064aac640 10 monclient: build_initial_monmap 2024-03-29T10:49:56.578 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.595+0000 7fb064aac640 10 monclient: monmap: 2024-03-29T10:49:56.578 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:49:56.578 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:56.578 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:49:56.596456+0000 2024-03-29T10:49:56.578 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:49:56.596456+0000 2024-03-29T10:49:56.578 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:56.578 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:56.578 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:49:56.578 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:49:56.578 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:49:56.578 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:56.579 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.597+0000 7fb064aac640 1 Processor -- start 2024-03-29T10:49:56.579 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.597+0000 7fb064aac640 1 -- start start 2024-03-29T10:49:56.579 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.597+0000 7fb064aac640 10 monclient: init 2024-03-29T10:49:56.579 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.598+0000 7fb064aac640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:56.579 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.598+0000 7fb064aac640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:49:56.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.598+0000 7fb064aac640 10 monclient(hunting): picked mon.noname-b con 0x7fb0580389e0 addr v1:172.21.15.57:6790/0 2024-03-29T10:49:56.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.598+0000 7fb064aac640 10 monclient(hunting): picked mon.noname-a con 0x7fb058038dc0 addr v1:172.21.15.57:6789/0 2024-03-29T10:49:56.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.598+0000 7fb064aac640 10 monclient(hunting): picked mon.noname-c con 0x7fb058030660 addr v1:172.21.15.179:6789/0 2024-03-29T10:49:56.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.598+0000 7fb064aac640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0580c54c0 con 0x7fb058038dc0 2024-03-29T10:49:56.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.598+0000 7fb064aac640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb058036430 con 0x7fb058030660 2024-03-29T10:49:56.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.598+0000 7fb064aac640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb058039840 con 0x7fb0580389e0 2024-03-29T10:49:56.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.598+0000 7fb064aac640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:56.580 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.598+0000 7fb064aac640 10 monclient(hunting): authenticate will time out at 1548.774216s 2024-03-29T10:49:56.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.601+0000 7fb05effd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fb058038dc0 0x7fb0580cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:53233/0 (socket says 172.21.15.57:53233) 2024-03-29T10:49:56.583 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.601+0000 7fb05effd640 1 -- 172.21.15.57:0/2593325509 learned_addr learned my addr 172.21.15.57:0/2593325509 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:49:56.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.604+0000 7fb05e7fc640 1 -- 172.21.15.57:0/2593325509 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1209496100 0 0) 0x7fb0580c54c0 con 0x7fb058038dc0 2024-03-29T10:49:56.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.604+0000 7fb05e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:49:56.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.604+0000 7fb05e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:56.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.604+0000 7fb05e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:56.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.604+0000 7fb05e7fc640 10 monclient(hunting): my global_id is 4569 2024-03-29T10:49:56.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.604+0000 7fb05e7fc640 1 -- 172.21.15.57:0/2593325509 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb050003520 con 0x7fb058038dc0 2024-03-29T10:49:56.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.605+0000 7fb05e7fc640 1 -- 172.21.15.57:0/2593325509 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 662610203 0 0) 0x7fb050003520 con 0x7fb058038dc0 2024-03-29T10:49:56.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.605+0000 7fb05e7fc640 1 -- 172.21.15.57:0/2593325509 >> v1:172.21.15.57:6790/0 conn(0x7fb0580389e0 legacy=0x7fb058033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:49:56.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.605+0000 7fb05e7fc640 1 -- 172.21.15.57:0/2593325509 >> v1:172.21.15.179:6789/0 conn(0x7fb058030660 legacy=0x7fb0580d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:56.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.605+0000 7fb05e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:49:56.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.605+0000 7fb05e7fc640 1 monclient: found mon.noname-a 2024-03-29T10:49:56.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.605+0000 7fb05e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:56.589 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.605+0000 7fb05e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:49:56.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.605+0000 7fb05e7fc640 1 -- 172.21.15.57:0/2593325509 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb058035690 con 0x7fb058038dc0 2024-03-29T10:49:56.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.605+0000 7fb05e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:49:56.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.605+0000 7fb05e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:49:56.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.605+0000 7fb05e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:56.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.605+0000 7fb05e7fc640 1 -- 172.21.15.57:0/2593325509 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb06004f730 con 0x7fb058038dc0 2024-03-29T10:49:56.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.605+0000 7fb064aac640 5 monclient: authenticate success, global_id 4569 2024-03-29T10:49:56.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.605+0000 7fb064aac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:56.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.605+0000 7fb05e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:56.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.605+0000 7fb05e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:49:56.590 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.605+0000 7fb05e7fc640 10 monclient: dump: 2024-03-29T10:49:56.590 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:56.590 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:56.590 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:56.590 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:56.591 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:56.591 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:56.591 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:56.591 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:56.591 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:56.591 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:56.591 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.605+0000 7fb064aac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:49:56.591 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.606+0000 7fb05e7fc640 1 -- 172.21.15.57:0/2593325509 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0600008d0 con 0x7fb058038dc0 2024-03-29T10:49:56.591 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.606+0000 7fb05e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:56.591 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.606+0000 7fb064aac640 10 monclient: get_monmap_and_config success 2024-03-29T10:49:56.591 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.606+0000 7fb064aac640 4 set_mon_vals no callback set 2024-03-29T10:49:56.591 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.606+0000 7fb05e7fc640 1 -- 172.21.15.57:0/2593325509 <== mon.0 v1:172.21.15.57:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb0600758b0 con 0x7fb058038dc0 2024-03-29T10:49:56.591 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.606+0000 7fb05e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:56.591 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.606+0000 7fb05e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:49:56.591 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.606+0000 7fb05e7fc640 10 monclient: dump: 2024-03-29T10:49:56.591 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:56.592 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:56.592 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:56.592 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:56.592 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:56.592 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:56.592 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:56.592 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:56.592 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:56.592 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:56.592 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.607+0000 7fb064aac640 10 monclient: shutdown 2024-03-29T10:49:56.592 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.607+0000 7fb064aac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:56.592 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.607+0000 7fb064aac640 1 -- 172.21.15.57:0/2593325509 >> v1:172.21.15.57:6789/0 conn(0x7fb058038dc0 legacy=0x7fb0580cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:56.592 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.607+0000 7fb064aac640 1 -- 172.21.15.57:0/2593325509 shutdown_connections 2024-03-29T10:49:56.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.607+0000 7fb064aac640 1 -- 172.21.15.57:0/2593325509 >> 172.21.15.57:0/2593325509 conn(0x7fb058005c00 msgr2=0x7fb058006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:56.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.609+0000 7fb064aac640 1 -- 172.21.15.57:0/2593325509 shutdown_connections 2024-03-29T10:49:56.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.611+0000 7fb064aac640 1 -- 172.21.15.57:0/2593325509 wait complete. 2024-03-29T10:49:56.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.612+0000 7fb064aac640 10 monclient: build_initial_monmap 2024-03-29T10:49:56.593 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.612+0000 7fb064aac640 10 monclient: monmap: 2024-03-29T10:49:56.594 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:49:56.594 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:49:56.594 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:49:56.594 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:49:56.594 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:49:56.594 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:56.594 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:49:56.594 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:49:56.594 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:49:56.594 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:56.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.612+0000 7fb064aac640 1 librados: starting msgr at 2024-03-29T10:49:56.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.612+0000 7fb064aac640 1 librados: starting objecter 2024-03-29T10:49:56.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.612+0000 7fb064aac640 1 Processor -- start 2024-03-29T10:49:56.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.613+0000 7fb064aac640 1 -- start start 2024-03-29T10:49:56.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.613+0000 7fb064aac640 1 librados: setting wanted keys 2024-03-29T10:49:56.595 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.613+0000 7fb064aac640 1 librados: calling monclient init 2024-03-29T10:49:56.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.613+0000 7fb064aac640 10 monclient: init 2024-03-29T10:49:56.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.613+0000 7fb064aac640 10 monclient: _reopen_session rank -1 2024-03-29T10:49:56.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.613+0000 7fb064aac640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:49:56.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.613+0000 7fb064aac640 10 monclient(hunting): picked mon.noname-a con 0x7fb058030660 addr v1:172.21.15.57:6789/0 2024-03-29T10:49:56.596 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.613+0000 7fb064aac640 10 monclient(hunting): picked mon.noname-b con 0x7fb0580389e0 addr v1:172.21.15.179:6789/0 2024-03-29T10:49:56.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.613+0000 7fb064aac640 10 monclient(hunting): picked mon.noname-c con 0x7fb05802d270 addr v1:172.21.15.57:6790/0 2024-03-29T10:49:56.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.616+0000 7fb064aac640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb05802f430 con 0x7fb058030660 2024-03-29T10:49:56.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.616+0000 7fb064aac640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb05802ec50 con 0x7fb0580389e0 2024-03-29T10:49:56.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.616+0000 7fb064aac640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb05802dd10 con 0x7fb05802d270 2024-03-29T10:49:56.597 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.616+0000 7fb064aac640 10 monclient(hunting): _renew_subs 2024-03-29T10:49:56.598 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.616+0000 7fb064aac640 10 monclient(hunting): authenticate will time out at 1548.792249s 2024-03-29T10:49:56.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.617+0000 7fb05f7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fb058030660 0x7fb0580c5650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:53517/0 (socket says 172.21.15.57:53517) 2024-03-29T10:49:56.600 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.617+0000 7fb05f7fe640 1 -- 172.21.15.57:0/3162996447 learned_addr learned my addr 172.21.15.57:0/3162996447 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:49:56.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.619+0000 7fb04bfff640 10 client.?.objecter ms_handle_connect 0x7fb058030660 2024-03-29T10:49:56.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.619+0000 7fb04bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:56.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.619+0000 7fb04bfff640 10 client.?.objecter ms_handle_connect 0x7fb0580389e0 2024-03-29T10:49:56.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.619+0000 7fb04bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:56.601 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.620+0000 7fb04bfff640 10 client.?.objecter ms_handle_connect 0x7fb05802d270 2024-03-29T10:49:56.602 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.620+0000 7fb04bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:49:56.602 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.621+0000 7fb04bfff640 1 -- 172.21.15.57:0/3162996447 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3571242236 0 0) 0x7fb05802f430 con 0x7fb058030660 2024-03-29T10:49:56.602 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.621+0000 7fb04bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:56.602 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.621+0000 7fb04bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:56.602 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.621+0000 7fb04bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:56.602 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.621+0000 7fb04bfff640 10 monclient(hunting): my global_id is 4572 2024-03-29T10:49:56.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.621+0000 7fb04bfff640 1 -- 172.21.15.57:0/3162996447 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb044004eb0 con 0x7fb058030660 2024-03-29T10:49:56.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.621+0000 7fb04bfff640 1 -- 172.21.15.57:0/3162996447 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734900809 0 0) 0x7fb05802ec50 con 0x7fb0580389e0 2024-03-29T10:49:56.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.621+0000 7fb04bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:49:56.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.621+0000 7fb04bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:49:56.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.621+0000 7fb04bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:49:56.603 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.621+0000 7fb04bfff640 10 monclient(hunting): my global_id is 4576 2024-03-29T10:49:56.604 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.621+0000 7fb04bfff640 1 -- 172.21.15.57:0/3162996447 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0440051d0 con 0x7fb0580389e0 2024-03-29T10:49:56.605 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb04bfff640 1 -- 172.21.15.57:0/3162996447 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4201769706 0 0) 0x7fb0440051d0 con 0x7fb0580389e0 2024-03-29T10:49:56.605 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb04bfff640 1 -- 172.21.15.57:0/3162996447 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb05802ec50 con 0x7fb0580389e0 2024-03-29T10:49:56.605 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb04bfff640 1 -- 172.21.15.57:0/3162996447 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb06004f400 con 0x7fb0580389e0 2024-03-29T10:49:56.605 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb04bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:56.605 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb04bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:49:56.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb04bfff640 10 monclient(hunting): dump: 2024-03-29T10:49:56.606 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:56.606 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:56.606 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:56.606 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:56.606 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:56.606 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:56.606 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:56.606 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:56.606 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:56.606 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:56.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb04bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:49:56.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb04bfff640 1 -- 172.21.15.57:0/3162996447 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 807785346 0 0) 0x7fb05802ec50 con 0x7fb0580389e0 2024-03-29T10:49:56.606 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb04bfff640 1 -- 172.21.15.57:0/3162996447 >> v1:172.21.15.57:6790/0 conn(0x7fb05802d270 legacy=0x7fb05816b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:56.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb04bfff640 1 -- 172.21.15.57:0/3162996447 >> v1:172.21.15.57:6789/0 conn(0x7fb058030660 legacy=0x7fb0580c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:56.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb04bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:49:56.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb04bfff640 1 monclient: found mon.b 2024-03-29T10:49:56.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb04bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:49:56.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb04bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:49:56.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb04bfff640 1 -- 172.21.15.57:0/3162996447 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0580c82c0 con 0x7fb0580389e0 2024-03-29T10:49:56.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb04bfff640 10 monclient: _finish_auth 0 2024-03-29T10:49:56.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb04bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:49:56.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb04bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:49:56.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb064aac640 5 monclient: authenticate success, global_id 4576 2024-03-29T10:49:56.607 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb064aac640 10 monclient: _renew_subs 2024-03-29T10:49:56.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb064aac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:49:56.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb064aac640 1 -- 172.21.15.57:0/3162996447 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0580c8570 con 0x7fb0580389e0 2024-03-29T10:49:56.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb064aac640 10 client.4576.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:49:56.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb064aac640 10 monclient: _renew_subs 2024-03-29T10:49:56.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb064aac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:49:56.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb064aac640 1 -- 172.21.15.57:0/3162996447 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb05816d270 con 0x7fb0580389e0 2024-03-29T10:49:56.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.623+0000 7fb064aac640 1 librados: init done 2024-03-29T10:49:56.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.624+0000 7fb04bfff640 1 -- 172.21.15.57:0/3162996447 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb06004fdd0 con 0x7fb0580389e0 2024-03-29T10:49:56.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.624+0000 7fb04bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:49:56.608 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.624+0000 7fb05dffb640 4 set_mon_vals no callback set 2024-03-29T10:49:56.609 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.624+0000 7fb04bfff640 1 -- 172.21.15.57:0/3162996447 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fb06004f400 con 0x7fb0580389e0 2024-03-29T10:49:56.609 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.624+0000 7fb04bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:49:56.609 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.624+0000 7fb04bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:49:56.609 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.624+0000 7fb04bfff640 10 monclient: dump: 2024-03-29T10:49:56.609 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:49:56.609 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:49:56.609 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:56.609 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:49:56.610 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:49:56.610 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:49:56.610 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:49:56.610 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:49:56.610 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:49:56.610 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:49:56.611 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.625+0000 7fb064aac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:56.611 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.626+0000 7fb04bfff640 1 -- 172.21.15.57:0/3162996447 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fb060067bf0 con 0x7fb0580389e0 2024-03-29T10:49:56.611 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.627+0000 7fb04bfff640 1 -- 172.21.15.57:0/3162996447 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3486612304 0 0) 0x7fb0600ceec0 con 0x7fb0580389e0 2024-03-29T10:49:56.611 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.627+0000 7fb04bfff640 10 client.4576.objecter ms_dispatch 0x7fb060000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:49:56.611 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.627+0000 7fb04bfff640 3 client.4576.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:49:56.612 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.627+0000 7fb04bfff640 3 client.4576.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:49:56.612 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.627+0000 7fb04bfff640 20 client.4576.objecter dump_active .. 0 homeless 2024-03-29T10:49:56.612 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.625+0000 7fb064aac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:56.612 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.627+0000 7fb064aac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:49:56.612 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.627+0000 7fb064aac640 1 -- 172.21.15.57:0/3162996447 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb040005080 con 0x7fb0580389e0 2024-03-29T10:49:56.612 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.627+0000 7fb05f7fe640 10 monclient: get_auth_request con 0x7fb04406e170 auth_method 0 2024-03-29T10:49:56.612 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.631+0000 7fb04bfff640 10 client.4576.objecter ms_handle_connect 0x7fb04406e170 2024-03-29T10:49:56.617 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.636+0000 7fb04bfff640 1 -- 172.21.15.57:0/3162996447 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb0600cb160 con 0x7fb0580389e0 2024-03-29T10:49:56.617 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.636+0000 7fb04bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:49:56.617 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.636+0000 7fb04bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:49:56.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.980+0000 7fb064aac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:56.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.980+0000 7fb064aac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:56.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.980+0000 7fb064aac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:49:56.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.980+0000 7fb064aac640 1 -- 172.21.15.57:0/3162996447 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb040005330 con 0x7fb0580389e0 2024-03-29T10:49:56.965 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.983+0000 7fb04bfff640 1 -- 172.21.15.57:0/3162996447 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2357246973) 0x7fb0600cb160 con 0x7fb0580389e0 2024-03-29T10:49:56.965 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.983+0000 7fb04bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:49:56.965 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.983+0000 7fb04bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:49:56.966 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:49:56.966 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":11,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:49:40.047000+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:49:40.047000+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:49:56.969 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.987+0000 7fb064aac640 10 librados: watch_flush enter 2024-03-29T10:49:56.969 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.987+0000 7fb064aac640 10 librados: watch_flush exit 2024-03-29T10:49:56.969 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.987+0000 7fb064aac640 20 client.4576.objecter shutdown clearing up homeless session... 2024-03-29T10:49:56.969 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.987+0000 7fb064aac640 10 client.4576.objecter successfully canceled tick 2024-03-29T10:49:56.970 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.988+0000 7fb064aac640 1 -- 172.21.15.57:0/3162996447 >> v1:172.21.15.179:6816/2698295936 conn(0x7fb04406e170 legacy=0x7fb044070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:56.970 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.988+0000 7fb064aac640 10 monclient: shutdown 2024-03-29T10:49:56.970 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.988+0000 7fb064aac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:49:56.970 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.988+0000 7fb064aac640 1 -- 172.21.15.57:0/3162996447 >> v1:172.21.15.179:6789/0 conn(0x7fb0580389e0 legacy=0x7fb05802fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:49:56.972 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.991+0000 7fb064aac640 1 -- 172.21.15.57:0/3162996447 shutdown_connections 2024-03-29T10:49:56.972 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.991+0000 7fb064aac640 1 -- 172.21.15.57:0/3162996447 >> 172.21.15.57:0/3162996447 conn(0x7fb058005c00 msgr2=0x7fb058034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:49:56.974 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.992+0000 7fb064aac640 1 -- 172.21.15.57:0/3162996447 shutdown_connections 2024-03-29T10:49:56.974 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.992+0000 7fb064aac640 1 -- 172.21.15.57:0/3162996447 wait complete. 2024-03-29T10:49:56.974 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:49:56.993+0000 7fb064aac640 1 librados: shutdown 2024-03-29T10:49:56.991 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:03.798 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:03.952 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.969+0000 7fecb9fe8640 10 monclient: get_monmap_and_config 2024-03-29T10:50:03.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.969+0000 7fecb9fe8640 10 monclient: build_initial_monmap 2024-03-29T10:50:03.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.969+0000 7fecb9fe8640 10 monclient: monmap: 2024-03-29T10:50:03.953 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:50:03.953 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:03.953 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:50:03.970305+0000 2024-03-29T10:50:03.953 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:50:03.970305+0000 2024-03-29T10:50:03.953 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:03.953 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:03.953 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:50:03.953 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:50:03.953 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:50:03.953 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:03.953 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.970+0000 7fecb9fe8640 1 Processor -- start 2024-03-29T10:50:03.954 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.970+0000 7fecb9fe8640 1 -- start start 2024-03-29T10:50:03.954 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.970+0000 7fecb9fe8640 10 monclient: init 2024-03-29T10:50:03.955 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.971+0000 7fecb9fe8640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:03.955 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.971+0000 7fecb9fe8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:50:03.955 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.971+0000 7fecb9fe8640 10 monclient(hunting): picked mon.noname-a con 0x7fecb40b0860 addr v1:172.21.15.57:6789/0 2024-03-29T10:50:03.955 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.971+0000 7fecb9fe8640 10 monclient(hunting): picked mon.noname-c con 0x7fecb413ef30 addr v1:172.21.15.179:6789/0 2024-03-29T10:50:03.955 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.971+0000 7fecb9fe8640 10 monclient(hunting): picked mon.noname-b con 0x7fecb40af8e0 addr v1:172.21.15.57:6790/0 2024-03-29T10:50:03.955 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.971+0000 7fecb9fe8640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecb40b0c40 con 0x7fecb40b0860 2024-03-29T10:50:03.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.971+0000 7fecb9fe8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecb40aed10 con 0x7fecb413ef30 2024-03-29T10:50:03.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.971+0000 7fecb9fe8640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecb40afd50 con 0x7fecb40af8e0 2024-03-29T10:50:03.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.971+0000 7fecb9fe8640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:03.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.971+0000 7fecb9fe8640 10 monclient(hunting): authenticate will time out at 1556.147875s 2024-03-29T10:50:03.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.974+0000 7fecb3fff640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7fecb40af8e0 0x7fecb4149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:48484/0 (socket says 172.21.15.57:48484) 2024-03-29T10:50:03.956 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.974+0000 7fecb3fff640 1 -- 172.21.15.57:0/530707376 learned_addr learned my addr 172.21.15.57:0/530707376 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:50:03.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.974+0000 7fecb27fc640 1 -- 172.21.15.57:0/530707376 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995717393 0 0) 0x7fecb40afd50 con 0x7fecb40af8e0 2024-03-29T10:50:03.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.974+0000 7fecb27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:03.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.974+0000 7fecb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:03.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.974+0000 7fecb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:03.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.974+0000 7fecb27fc640 10 monclient(hunting): my global_id is 4640 2024-03-29T10:50:03.957 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.974+0000 7fecb27fc640 1 -- 172.21.15.57:0/530707376 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec9c003500 con 0x7fecb40af8e0 2024-03-29T10:50:03.958 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.976+0000 7fecb27fc640 1 -- 172.21.15.57:0/530707376 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2996794434 0 0) 0x7fecb40aed10 con 0x7fecb413ef30 2024-03-29T10:50:03.958 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.976+0000 7fecb27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:03.958 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.976+0000 7fecb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:03.958 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.976+0000 7fecb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:03.958 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.976+0000 7fecb27fc640 10 monclient(hunting): my global_id is 4579 2024-03-29T10:50:03.958 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.976+0000 7fecb27fc640 1 -- 172.21.15.57:0/530707376 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecb40afd50 con 0x7fecb413ef30 2024-03-29T10:50:03.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.978+0000 7fecb27fc640 1 -- 172.21.15.57:0/530707376 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2398372229 0 0) 0x7fec9c003500 con 0x7fecb40af8e0 2024-03-29T10:50:03.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.978+0000 7fecb27fc640 1 -- 172.21.15.57:0/530707376 >> v1:172.21.15.179:6789/0 conn(0x7fecb413ef30 legacy=0x7fecb40af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:03.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.978+0000 7fecb27fc640 1 -- 172.21.15.57:0/530707376 >> v1:172.21.15.57:6789/0 conn(0x7fecb40b0860 legacy=0x7fecb413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:03.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.978+0000 7fecb27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:50:03.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.978+0000 7fecb27fc640 1 monclient: found mon.noname-b 2024-03-29T10:50:03.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.978+0000 7fecb27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:03.960 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.978+0000 7fecb27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:50:03.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.978+0000 7fecb27fc640 1 -- 172.21.15.57:0/530707376 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb40ab1b0 con 0x7fecb40af8e0 2024-03-29T10:50:03.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.978+0000 7fecb27fc640 10 monclient: _finish_auth 0 2024-03-29T10:50:03.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.978+0000 7fecb27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:50:03.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.978+0000 7fecb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:03.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.978+0000 7fecb27fc640 1 -- 172.21.15.57:0/530707376 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7feca8004cb0 con 0x7fecb40af8e0 2024-03-29T10:50:03.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.978+0000 7fecb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:03.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.978+0000 7fecb27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:03.961 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.978+0000 7fecb27fc640 10 monclient: dump: 2024-03-29T10:50:03.961 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:50:03.961 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:03.961 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:03.962 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:03.962 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:03.962 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:03.962 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:50:03.962 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:50:03.962 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:50:03.962 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:03.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.978+0000 7fecb27fc640 1 -- 172.21.15.57:0/530707376 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feca8004e10 con 0x7fecb40af8e0 2024-03-29T10:50:03.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.978+0000 7fecb27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:03.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.978+0000 7fecb27fc640 1 -- 172.21.15.57:0/530707376 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7feca80013f0 con 0x7fecb40af8e0 2024-03-29T10:50:03.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.978+0000 7fecb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:03.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.978+0000 7fecb27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:03.962 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.978+0000 7fecb27fc640 10 monclient: dump: 2024-03-29T10:50:03.962 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:50:03.963 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:03.963 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:03.963 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:03.963 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:03.963 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:03.963 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:50:03.963 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:50:03.963 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:50:03.963 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:03.963 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.978+0000 7fecb9fe8640 5 monclient: authenticate success, global_id 4640 2024-03-29T10:50:03.963 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.978+0000 7fecb9fe8640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:03.964 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.978+0000 7fecb9fe8640 4 set_mon_vals no callback set 2024-03-29T10:50:03.965 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.980+0000 7fecb9fe8640 10 monclient: shutdown 2024-03-29T10:50:03.965 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.980+0000 7fecb9fe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:03.965 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.980+0000 7fecb9fe8640 1 -- 172.21.15.57:0/530707376 >> v1:172.21.15.57:6790/0 conn(0x7fecb40af8e0 legacy=0x7fecb4149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:03.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.982+0000 7fecb9fe8640 1 -- 172.21.15.57:0/530707376 shutdown_connections 2024-03-29T10:50:03.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.983+0000 7fecb9fe8640 1 -- 172.21.15.57:0/530707376 >> 172.21.15.57:0/530707376 conn(0x7fecb406b660 msgr2=0x7fecb406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:03.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.983+0000 7fecb9fe8640 1 -- 172.21.15.57:0/530707376 shutdown_connections 2024-03-29T10:50:03.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.983+0000 7fecb9fe8640 1 -- 172.21.15.57:0/530707376 wait complete. 2024-03-29T10:50:03.966 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.984+0000 7fecb9fe8640 10 monclient: build_initial_monmap 2024-03-29T10:50:03.967 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.984+0000 7fecb9fe8640 10 monclient: monmap: 2024-03-29T10:50:03.967 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:50:03.967 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:03.967 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:50:03.967 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:50:03.967 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:03.967 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:03.967 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:50:03.967 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:50:03.967 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:50:03.967 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:03.967 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.984+0000 7fecb9fe8640 1 librados: starting msgr at 2024-03-29T10:50:03.967 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.984+0000 7fecb9fe8640 1 librados: starting objecter 2024-03-29T10:50:03.967 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.984+0000 7fecb9fe8640 1 Processor -- start 2024-03-29T10:50:03.969 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.987+0000 7fecb9fe8640 1 -- start start 2024-03-29T10:50:03.969 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.987+0000 7fecb9fe8640 1 librados: setting wanted keys 2024-03-29T10:50:03.969 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.987+0000 7fecb9fe8640 1 librados: calling monclient init 2024-03-29T10:50:03.969 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.987+0000 7fecb9fe8640 10 monclient: init 2024-03-29T10:50:03.972 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.988+0000 7fecb9fe8640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:03.972 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.988+0000 7fecb9fe8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:50:03.972 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.988+0000 7fecb9fe8640 10 monclient(hunting): picked mon.noname-b con 0x7fecb40af8e0 addr v1:172.21.15.179:6789/0 2024-03-29T10:50:03.972 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.988+0000 7fecb9fe8640 10 monclient(hunting): picked mon.noname-c con 0x7fecb40b0860 addr v1:172.21.15.57:6790/0 2024-03-29T10:50:03.972 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.988+0000 7fecb9fe8640 10 monclient(hunting): picked mon.noname-a con 0x7fecb413ef30 addr v1:172.21.15.57:6789/0 2024-03-29T10:50:03.972 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.988+0000 7fecb9fe8640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecb40b26a0 con 0x7fecb413ef30 2024-03-29T10:50:03.972 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.988+0000 7fecb9fe8640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecb40b2f40 con 0x7fecb40af8e0 2024-03-29T10:50:03.972 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.988+0000 7fecb9fe8640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecb41384d0 con 0x7fecb40b0860 2024-03-29T10:50:03.972 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.988+0000 7fecb9fe8640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:03.972 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.988+0000 7fecb9fe8640 10 monclient(hunting): authenticate will time out at 1556.164564s 2024-03-29T10:50:03.972 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.988+0000 7fecb2ffd640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7fecb40b0860 0x7fecb40b3b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:49164/0 (socket says 172.21.15.57:49164) 2024-03-29T10:50:03.973 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.988+0000 7fecb2ffd640 1 -- 172.21.15.57:0/1169355088 learned_addr learned my addr 172.21.15.57:0/1169355088 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:50:03.973 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.990+0000 7fec9bfff640 10 client.?.objecter ms_handle_connect 0x7fecb40b0860 2024-03-29T10:50:03.973 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.990+0000 7fec9bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:03.973 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.990+0000 7fec9bfff640 1 -- 172.21.15.57:0/1169355088 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192717669 0 0) 0x7fecb41384d0 con 0x7fecb40b0860 2024-03-29T10:50:03.973 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.990+0000 7fec9bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:03.973 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.990+0000 7fec9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:03.973 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.990+0000 7fec9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:03.973 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.990+0000 7fec9bfff640 10 monclient(hunting): my global_id is 4643 2024-03-29T10:50:03.973 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.990+0000 7fec9bfff640 1 -- 172.21.15.57:0/1169355088 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feca0004eb0 con 0x7fecb40b0860 2024-03-29T10:50:03.976 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.991+0000 7fec9bfff640 1 -- 172.21.15.57:0/1169355088 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1559392997 0 0) 0x7feca0004eb0 con 0x7fecb40b0860 2024-03-29T10:50:03.976 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.991+0000 7fec9bfff640 1 -- 172.21.15.57:0/1169355088 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecb41384d0 con 0x7fecb40b0860 2024-03-29T10:50:03.976 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.991+0000 7fec9bfff640 1 -- 172.21.15.57:0/1169355088 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fecac001cf0 con 0x7fecb40b0860 2024-03-29T10:50:03.976 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.991+0000 7fec9bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:03.976 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.991+0000 7fec9bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:03.976 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.991+0000 7fec9bfff640 10 monclient(hunting): dump: 2024-03-29T10:50:03.976 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:50:03.976 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:03.976 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:03.976 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:03.976 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:03.976 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:03.977 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:50:03.977 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:50:03.977 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:50:03.977 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:03.977 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.991+0000 7fec9bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:03.977 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.991+0000 7fec9bfff640 1 -- 172.21.15.57:0/1169355088 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 189214591 0 0) 0x7fecb41384d0 con 0x7fecb40b0860 2024-03-29T10:50:03.977 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.992+0000 7fec9bfff640 1 -- 172.21.15.57:0/1169355088 >> v1:172.21.15.179:6789/0 conn(0x7fecb40af8e0 legacy=0x7fecb40b3460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:50:03.977 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.993+0000 7fec9bfff640 1 -- 172.21.15.57:0/1169355088 >> v1:172.21.15.57:6789/0 conn(0x7fecb413ef30 legacy=0x7fecb41dd9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:03.977 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.993+0000 7fec9bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:50:03.977 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.993+0000 7fec9bfff640 1 monclient: found mon.c 2024-03-29T10:50:03.977 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.993+0000 7fec9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:03.977 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.993+0000 7fec9bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:50:03.977 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.993+0000 7fec9bfff640 1 -- 172.21.15.57:0/1169355088 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb4138660 con 0x7fecb40b0860 2024-03-29T10:50:03.977 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.993+0000 7fec9bfff640 10 monclient: _finish_auth 0 2024-03-29T10:50:03.977 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.993+0000 7fec9bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:50:03.978 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.993+0000 7fec9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:03.978 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.993+0000 7fecb9fe8640 5 monclient: authenticate success, global_id 4643 2024-03-29T10:50:03.978 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.993+0000 7fecb9fe8640 10 monclient: _renew_subs 2024-03-29T10:50:03.978 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.993+0000 7fecb9fe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:50:03.978 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.993+0000 7fecb9fe8640 1 -- 172.21.15.57:0/1169355088 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecb4137a30 con 0x7fecb40b0860 2024-03-29T10:50:03.978 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.993+0000 7fecb9fe8640 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:03.978 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.993+0000 7fecb9fe8640 10 monclient: _renew_subs 2024-03-29T10:50:03.978 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.993+0000 7fecb9fe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:50:03.978 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.993+0000 7fecb9fe8640 1 -- 172.21.15.57:0/1169355088 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fecb40b1440 con 0x7fecb40b0860 2024-03-29T10:50:03.978 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.994+0000 7fecb9fe8640 1 librados: init done 2024-03-29T10:50:03.979 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.996+0000 7fecb9fe8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:03.979 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.996+0000 7fecb9fe8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:03.980 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.996+0000 7fecb9fe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:50:03.980 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.996+0000 7fecb9fe8640 1 -- 172.21.15.57:0/1169355088 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecb4075f10 con 0x7fecb40b0860 2024-03-29T10:50:03.980 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.996+0000 7fec9bfff640 1 -- 172.21.15.57:0/1169355088 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecac001e50 con 0x7fecb40b0860 2024-03-29T10:50:03.980 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.996+0000 7fec9bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:03.980 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.996+0000 7fec9bfff640 1 -- 172.21.15.57:0/1169355088 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fecac003de0 con 0x7fecb40b0860 2024-03-29T10:50:03.980 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.996+0000 7fec9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:03.980 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.996+0000 7fec9bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:03.980 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.996+0000 7fec9bfff640 10 monclient: dump: 2024-03-29T10:50:03.980 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:50:03.980 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:03.980 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:03.980 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:03.980 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:03.980 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:03.980 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:50:03.981 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:50:03.981 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:50:03.981 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:03.981 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.997+0000 7fecb17fa640 4 set_mon_vals no callback set 2024-03-29T10:50:03.981 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.997+0000 7fec9bfff640 1 -- 172.21.15.57:0/1169355088 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fecac004000 con 0x7fecb40b0860 2024-03-29T10:50:03.982 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:03.999+0000 7fecb37fe640 10 monclient: get_auth_request con 0x7feca006dcd0 auth_method 0 2024-03-29T10:50:03.982 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.000+0000 7fec9bfff640 1 -- 172.21.15.57:0/1169355088 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3486612304 0 0) 0x7fecac084e20 con 0x7fecb40b0860 2024-03-29T10:50:03.982 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.000+0000 7fec9bfff640 10 client.4643.objecter ms_dispatch 0x7fecb4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:50:03.982 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.000+0000 7fec9bfff640 3 client.4643.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:50:03.982 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.000+0000 7fec9bfff640 3 client.4643.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:50:03.982 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.000+0000 7fec9bfff640 20 client.4643.objecter dump_active .. 0 homeless 2024-03-29T10:50:03.982 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.000+0000 7fec9bfff640 10 client.4643.objecter ms_handle_connect 0x7feca006dcd0 2024-03-29T10:50:03.989 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.007+0000 7fec9bfff640 1 -- 172.21.15.57:0/1169355088 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fecac050700 con 0x7fecb40b0860 2024-03-29T10:50:03.989 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.007+0000 7fec9bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:03.989 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.007+0000 7fec9bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:04.324 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.343+0000 7fecb9fe8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:04.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.343+0000 7fecb9fe8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:04.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.343+0000 7fecb9fe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:50:04.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.343+0000 7fecb9fe8640 1 -- 172.21.15.57:0/1169355088 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fecb4072330 con 0x7fecb40b0860 2024-03-29T10:50:04.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.343+0000 7fec9bfff640 1 -- 172.21.15.57:0/1169355088 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2357246973) 0x7fecac052630 con 0x7fecb40b0860 2024-03-29T10:50:04.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.343+0000 7fec9bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:04.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.343+0000 7fec9bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:04.326 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:50:04.326 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":11,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:49:40.047000+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:49:40.047000+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:04.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.347+0000 7fecb9fe8640 10 librados: watch_flush enter 2024-03-29T10:50:04.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.347+0000 7fecb9fe8640 10 librados: watch_flush exit 2024-03-29T10:50:04.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.347+0000 7fecb9fe8640 20 client.4643.objecter shutdown clearing up homeless session... 2024-03-29T10:50:04.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.347+0000 7fecb9fe8640 10 client.4643.objecter successfully canceled tick 2024-03-29T10:50:04.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.347+0000 7fecb9fe8640 1 -- 172.21.15.57:0/1169355088 >> v1:172.21.15.179:6816/2698295936 conn(0x7feca006dcd0 legacy=0x7feca0070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:04.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.347+0000 7fecb9fe8640 10 monclient: shutdown 2024-03-29T10:50:04.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.347+0000 7fecb9fe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:04.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.347+0000 7fecb9fe8640 1 -- 172.21.15.57:0/1169355088 >> v1:172.21.15.57:6790/0 conn(0x7fecb40b0860 legacy=0x7fecb40b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:04.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.348+0000 7fecb9fe8640 1 -- 172.21.15.57:0/1169355088 shutdown_connections 2024-03-29T10:50:04.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.348+0000 7fecb9fe8640 1 -- 172.21.15.57:0/1169355088 >> 172.21.15.57:0/1169355088 conn(0x7fecb406b660 msgr2=0x7fecb406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:04.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.351+0000 7fecb9fe8640 1 -- 172.21.15.57:0/1169355088 shutdown_connections 2024-03-29T10:50:04.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.351+0000 7fecb9fe8640 1 -- 172.21.15.57:0/1169355088 wait complete. 2024-03-29T10:50:04.335 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:04.353+0000 7fecb9fe8640 1 librados: shutdown 2024-03-29T10:50:04.353 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:11.160 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:11.316 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.335+0000 7feb2a082640 10 monclient: get_monmap_and_config 2024-03-29T10:50:11.317 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.335+0000 7feb2a082640 10 monclient: build_initial_monmap 2024-03-29T10:50:11.317 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.335+0000 7feb2a082640 10 monclient: monmap: 2024-03-29T10:50:11.317 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:50:11.317 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:11.317 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:50:11.335922+0000 2024-03-29T10:50:11.317 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:50:11.335922+0000 2024-03-29T10:50:11.317 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:11.317 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:11.317 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:50:11.317 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:50:11.317 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:50:11.318 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:11.318 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.336+0000 7feb2a082640 1 Processor -- start 2024-03-29T10:50:11.318 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.336+0000 7feb2a082640 1 -- start start 2024-03-29T10:50:11.318 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.336+0000 7feb2a082640 10 monclient: init 2024-03-29T10:50:11.319 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.337+0000 7feb2a082640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:11.319 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.337+0000 7feb2a082640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:50:11.319 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.337+0000 7feb2a082640 10 monclient(hunting): picked mon.noname-c con 0x7feb240af290 addr v1:172.21.15.179:6789/0 2024-03-29T10:50:11.319 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.337+0000 7feb2a082640 10 monclient(hunting): picked mon.noname-a con 0x7feb24143ec0 addr v1:172.21.15.57:6789/0 2024-03-29T10:50:11.319 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.337+0000 7feb2a082640 10 monclient(hunting): picked mon.noname-b con 0x7feb240af670 addr v1:172.21.15.57:6790/0 2024-03-29T10:50:11.319 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.337+0000 7feb2a082640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb240ae740 con 0x7feb24143ec0 2024-03-29T10:50:11.319 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.337+0000 7feb2a082640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb240b06a0 con 0x7feb240af290 2024-03-29T10:50:11.319 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.337+0000 7feb2a082640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb240b0980 con 0x7feb240af670 2024-03-29T10:50:11.319 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.337+0000 7feb2a082640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:11.319 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.337+0000 7feb2a082640 10 monclient(hunting): authenticate will time out at 1563.513407s 2024-03-29T10:50:11.319 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.337+0000 7feb237fe640 1 --1- >> v1:172.21.15.179:6789/0 conn(0x7feb240af290 0x7feb241437b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.179:6789/0 says I am v1:172.21.15.57:50608/0 (socket says 172.21.15.57:50608) 2024-03-29T10:50:11.319 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.337+0000 7feb237fe640 1 -- 172.21.15.57:0/4285727052 learned_addr learned my addr 172.21.15.57:0/4285727052 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:50:11.320 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 1 -- 172.21.15.57:0/4285727052 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483895836 0 0) 0x7feb240b06a0 con 0x7feb240af290 2024-03-29T10:50:11.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:11.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:11.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:11.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 10 monclient(hunting): my global_id is 4582 2024-03-29T10:50:11.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 1 -- 172.21.15.57:0/4285727052 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb0c003520 con 0x7feb240af290 2024-03-29T10:50:11.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 1 -- 172.21.15.57:0/4285727052 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940068239 0 0) 0x7feb240b0980 con 0x7feb240af670 2024-03-29T10:50:11.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:11.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:11.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:11.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 10 monclient(hunting): my global_id is 4646 2024-03-29T10:50:11.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 1 -- 172.21.15.57:0/4285727052 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb240b06a0 con 0x7feb240af670 2024-03-29T10:50:11.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 1 -- 172.21.15.57:0/4285727052 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2247971795 0 0) 0x7feb0c003520 con 0x7feb240af290 2024-03-29T10:50:11.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 1 -- 172.21.15.57:0/4285727052 >> v1:172.21.15.57:6790/0 conn(0x7feb240af670 legacy=0x7feb24151dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:11.321 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 1 -- 172.21.15.57:0/4285727052 >> v1:172.21.15.57:6789/0 conn(0x7feb24143ec0 legacy=0x7feb240b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:11.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 10 monclient: _finish_hunting 0 2024-03-29T10:50:11.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 1 monclient: found mon.noname-c 2024-03-29T10:50:11.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:11.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:50:11.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 1 -- 172.21.15.57:0/4285727052 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb240b0c60 con 0x7feb240af290 2024-03-29T10:50:11.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 10 monclient: _finish_auth 0 2024-03-29T10:50:11.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 10 monclient: _check_auth_tickets 2024-03-29T10:50:11.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:11.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb2a082640 5 monclient: authenticate success, global_id 4582 2024-03-29T10:50:11.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb2a082640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:11.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 1 -- 172.21.15.57:0/4285727052 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7feb1c001a10 con 0x7feb240af290 2024-03-29T10:50:11.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:11.322 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:11.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 10 monclient: dump: 2024-03-29T10:50:11.323 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:50:11.323 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:11.323 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:11.323 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:11.323 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:11.323 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:11.323 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:50:11.323 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:50:11.323 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:50:11.323 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:11.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb2a082640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:11.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 1 -- 172.21.15.57:0/4285727052 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb1c001bf0 con 0x7feb240af290 2024-03-29T10:50:11.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:11.323 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 1 -- 172.21.15.57:0/4285727052 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7feb1c001a10 con 0x7feb240af290 2024-03-29T10:50:11.324 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:11.324 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:11.324 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.338+0000 7feb227fc640 10 monclient: dump: 2024-03-29T10:50:11.324 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:50:11.324 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:11.324 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:11.324 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:11.324 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:11.324 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:11.324 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:50:11.324 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:50:11.324 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:50:11.324 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:11.324 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.339+0000 7feb2a082640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:11.325 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.339+0000 7feb2a082640 4 set_mon_vals no callback set 2024-03-29T10:50:11.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.339+0000 7feb2a082640 10 monclient: shutdown 2024-03-29T10:50:11.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.339+0000 7feb2a082640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:11.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.339+0000 7feb2a082640 1 -- 172.21.15.57:0/4285727052 >> v1:172.21.15.179:6789/0 conn(0x7feb240af290 legacy=0x7feb241437b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:11.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.339+0000 7feb2a082640 1 -- 172.21.15.57:0/4285727052 shutdown_connections 2024-03-29T10:50:11.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.339+0000 7feb2a082640 1 -- 172.21.15.57:0/4285727052 >> 172.21.15.57:0/4285727052 conn(0x7feb24066a00 msgr2=0x7feb24066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:11.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.340+0000 7feb2a082640 1 -- 172.21.15.57:0/4285727052 shutdown_connections 2024-03-29T10:50:11.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.340+0000 7feb2a082640 1 -- 172.21.15.57:0/4285727052 wait complete. 2024-03-29T10:50:11.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.341+0000 7feb2a082640 10 monclient: build_initial_monmap 2024-03-29T10:50:11.326 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.341+0000 7feb2a082640 10 monclient: monmap: 2024-03-29T10:50:11.327 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:50:11.327 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:11.327 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:50:11.327 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:50:11.327 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:11.327 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:11.327 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:50:11.327 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:50:11.327 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:50:11.327 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:11.327 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.342+0000 7feb2a082640 1 librados: starting msgr at 2024-03-29T10:50:11.327 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.342+0000 7feb2a082640 1 librados: starting objecter 2024-03-29T10:50:11.327 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.342+0000 7feb2a082640 1 Processor -- start 2024-03-29T10:50:11.327 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.342+0000 7feb2a082640 1 -- start start 2024-03-29T10:50:11.327 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.342+0000 7feb2a082640 1 librados: setting wanted keys 2024-03-29T10:50:11.328 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.342+0000 7feb2a082640 1 librados: calling monclient init 2024-03-29T10:50:11.328 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.342+0000 7feb2a082640 10 monclient: init 2024-03-29T10:50:11.328 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.343+0000 7feb2a082640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:11.328 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.343+0000 7feb2a082640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:50:11.328 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.343+0000 7feb2a082640 10 monclient(hunting): picked mon.noname-c con 0x7feb240af290 addr v1:172.21.15.57:6790/0 2024-03-29T10:50:11.328 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.343+0000 7feb2a082640 10 monclient(hunting): picked mon.noname-b con 0x7feb240af670 addr v1:172.21.15.179:6789/0 2024-03-29T10:50:11.328 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.343+0000 7feb237fe640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7feb240af290 0x7feb24145e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:57114/0 (socket says 172.21.15.57:57114) 2024-03-29T10:50:11.328 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb237fe640 1 -- 172.21.15.57:0/4174726756 learned_addr learned my addr 172.21.15.57:0/4174726756 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:50:11.328 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb2a082640 10 monclient(hunting): picked mon.noname-a con 0x7feb24143ec0 addr v1:172.21.15.57:6789/0 2024-03-29T10:50:11.328 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb0bfff640 10 client.?.objecter ms_handle_connect 0x7feb240af290 2024-03-29T10:50:11.328 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb0bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:11.328 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb2a082640 1 -- 172.21.15.57:0/4174726756 --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb24146980 con 0x7feb24143ec0 2024-03-29T10:50:11.328 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb2a082640 1 -- 172.21.15.57:0/4174726756 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb241456d0 con 0x7feb240af670 2024-03-29T10:50:11.328 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb0bfff640 10 client.?.objecter ms_handle_connect 0x7feb24143ec0 2024-03-29T10:50:11.328 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb0bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:11.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb2a082640 1 -- 172.21.15.57:0/4174726756 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb24146580 con 0x7feb240af290 2024-03-29T10:50:11.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb2a082640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:11.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb2a082640 10 monclient(hunting): authenticate will time out at 1563.520430s 2024-03-29T10:50:11.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb0bfff640 10 client.?.objecter ms_handle_connect 0x7feb240af670 2024-03-29T10:50:11.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb0bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:11.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689318041 0 0) 0x7feb24146980 con 0x7feb24143ec0 2024-03-29T10:50:11.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb0bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:11.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:11.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:11.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb0bfff640 10 monclient(hunting): my global_id is 4581 2024-03-29T10:50:11.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb10004eb0 con 0x7feb24143ec0 2024-03-29T10:50:11.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626685935 0 0) 0x7feb241456d0 con 0x7feb240af670 2024-03-29T10:50:11.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb0bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:11.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:11.329 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:11.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb0bfff640 10 monclient(hunting): my global_id is 4585 2024-03-29T10:50:11.330 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.344+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb24146980 con 0x7feb240af670 2024-03-29T10:50:11.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.345+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 298564944 0 0) 0x7feb10004eb0 con 0x7feb24143ec0 2024-03-29T10:50:11.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.345+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb241456d0 con 0x7feb24143ec0 2024-03-29T10:50:11.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.345+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7feb14001930 con 0x7feb24143ec0 2024-03-29T10:50:11.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.345+0000 7feb0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:11.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.345+0000 7feb0bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:11.332 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.345+0000 7feb0bfff640 10 monclient(hunting): dump: 2024-03-29T10:50:11.332 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:50:11.332 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:11.332 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:11.332 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:11.332 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:11.332 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:11.333 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:50:11.333 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:50:11.333 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:50:11.333 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:11.333 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.345+0000 7feb0bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:11.333 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.345+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2362874481 0 0) 0x7feb24146980 con 0x7feb240af670 2024-03-29T10:50:11.333 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.345+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb10004eb0 con 0x7feb240af670 2024-03-29T10:50:11.333 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.345+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7feb18001800 con 0x7feb240af670 2024-03-29T10:50:11.333 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.345+0000 7feb0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:11.333 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.345+0000 7feb0bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:11.333 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.345+0000 7feb0bfff640 10 monclient(hunting): dump: 2024-03-29T10:50:11.333 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:50:11.333 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:11.333 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:11.334 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:11.334 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:11.334 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:11.334 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:50:11.334 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:50:11.334 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:50:11.334 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:11.334 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.345+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453487271 0 0) 0x7feb24146580 con 0x7feb240af290 2024-03-29T10:50:11.334 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.345+0000 7feb0bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:11.334 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.345+0000 7feb0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:11.334 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.345+0000 7feb0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:11.334 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.345+0000 7feb0bfff640 10 monclient(hunting): my global_id is 4649 2024-03-29T10:50:11.334 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.345+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb24146980 con 0x7feb240af290 2024-03-29T10:50:11.334 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.346+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 932793398 0 0) 0x7feb24146980 con 0x7feb240af290 2024-03-29T10:50:11.335 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.346+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb24146580 con 0x7feb240af290 2024-03-29T10:50:11.335 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.346+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7feb1c001a10 con 0x7feb240af290 2024-03-29T10:50:11.335 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.346+0000 7feb0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:11.335 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.346+0000 7feb0bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:11.335 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.346+0000 7feb0bfff640 10 monclient(hunting): dump: 2024-03-29T10:50:11.335 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:50:11.335 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:11.335 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:11.335 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:11.335 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:11.335 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:11.335 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:50:11.335 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:50:11.335 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:50:11.336 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:11.336 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.346+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2229067586 0 0) 0x7feb10004eb0 con 0x7feb240af670 2024-03-29T10:50:11.336 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.347+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 >> v1:172.21.15.57:6790/0 conn(0x7feb240af290 legacy=0x7feb24145e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:11.336 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.347+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 >> v1:172.21.15.57:6789/0 conn(0x7feb24143ec0 legacy=0x7feb241db520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:11.336 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.347+0000 7feb0bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:50:11.336 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.347+0000 7feb0bfff640 1 monclient: found mon.b 2024-03-29T10:50:11.336 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.347+0000 7feb0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:11.336 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.347+0000 7feb0bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:50:11.336 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.347+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb241d0750 con 0x7feb240af670 2024-03-29T10:50:11.336 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.347+0000 7feb0bfff640 10 monclient: _finish_auth 0 2024-03-29T10:50:11.336 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.347+0000 7feb0bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:50:11.336 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.347+0000 7feb0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:11.336 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.347+0000 7feb2a082640 5 monclient: authenticate success, global_id 4585 2024-03-29T10:50:11.336 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.347+0000 7feb2a082640 10 monclient: _renew_subs 2024-03-29T10:50:11.336 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.347+0000 7feb2a082640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:50:11.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.347+0000 7feb2a082640 1 -- 172.21.15.57:0/4174726756 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb241d0a00 con 0x7feb240af670 2024-03-29T10:50:11.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.347+0000 7feb2a082640 10 client.4585.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:11.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.347+0000 7feb2a082640 10 monclient: _renew_subs 2024-03-29T10:50:11.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.347+0000 7feb2a082640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:50:11.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.347+0000 7feb2a082640 1 -- 172.21.15.57:0/4174726756 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2413edb0 con 0x7feb240af670 2024-03-29T10:50:11.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.347+0000 7feb2a082640 1 librados: init done 2024-03-29T10:50:11.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.347+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb18001cb0 con 0x7feb240af670 2024-03-29T10:50:11.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.347+0000 7feb0bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:11.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.347+0000 7feb21ffb640 4 set_mon_vals no callback set 2024-03-29T10:50:11.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.348+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7feb18003790 con 0x7feb240af670 2024-03-29T10:50:11.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.348+0000 7feb0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:11.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.348+0000 7feb0bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:11.337 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.348+0000 7feb0bfff640 10 monclient: dump: 2024-03-29T10:50:11.337 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:50:11.337 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:11.338 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:11.338 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:11.338 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:11.338 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:11.338 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:50:11.338 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:50:11.338 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:50:11.338 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:11.338 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.348+0000 7feb2a082640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:11.338 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.348+0000 7feb2a082640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:11.338 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.350+0000 7feb2a082640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:50:11.338 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.350+0000 7feb2a082640 1 -- 172.21.15.57:0/4174726756 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb04005080 con 0x7feb240af670 2024-03-29T10:50:11.338 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.350+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7feb18003970 con 0x7feb240af670 2024-03-29T10:50:11.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.350+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3486612304 0 0) 0x7feb18083710 con 0x7feb240af670 2024-03-29T10:50:11.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.350+0000 7feb237fe640 10 monclient: get_auth_request con 0x7feb1006e0a0 auth_method 0 2024-03-29T10:50:11.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.350+0000 7feb0bfff640 10 client.4585.objecter ms_dispatch 0x7feb24000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:50:11.339 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.350+0000 7feb0bfff640 3 client.4585.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:50:11.340 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.350+0000 7feb0bfff640 3 client.4585.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:50:11.340 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.351+0000 7feb0bfff640 20 client.4585.objecter dump_active .. 0 homeless 2024-03-29T10:50:11.340 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.351+0000 7feb0bfff640 10 client.4585.objecter ms_handle_connect 0x7feb1006e0a0 2024-03-29T10:50:11.340 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.357+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feb1804ee20 con 0x7feb240af670 2024-03-29T10:50:11.340 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.357+0000 7feb0bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:11.340 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.357+0000 7feb0bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:11.685 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.704+0000 7feb2a082640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:11.686 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.704+0000 7feb2a082640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:11.686 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.704+0000 7feb2a082640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:50:11.686 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.704+0000 7feb2a082640 1 -- 172.21.15.57:0/4174726756 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb04005330 con 0x7feb240af670 2024-03-29T10:50:11.686 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.705+0000 7feb0bfff640 1 -- 172.21.15.57:0/4174726756 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2357246973) 0x7feb18050f20 con 0x7feb240af670 2024-03-29T10:50:11.686 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.705+0000 7feb0bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:11.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.705+0000 7feb0bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:11.687 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:50:11.687 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":11,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:49:40.047000+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:49:40.047000+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:11.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.708+0000 7feb2a082640 10 librados: watch_flush enter 2024-03-29T10:50:11.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.708+0000 7feb2a082640 10 librados: watch_flush exit 2024-03-29T10:50:11.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.708+0000 7feb2a082640 20 client.4585.objecter shutdown clearing up homeless session... 2024-03-29T10:50:11.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.708+0000 7feb2a082640 10 client.4585.objecter successfully canceled tick 2024-03-29T10:50:11.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.708+0000 7feb2a082640 1 -- 172.21.15.57:0/4174726756 >> v1:172.21.15.179:6816/2698295936 conn(0x7feb1006e0a0 legacy=0x7feb10070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:11.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.708+0000 7feb2a082640 10 monclient: shutdown 2024-03-29T10:50:11.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.708+0000 7feb2a082640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:11.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.708+0000 7feb2a082640 1 -- 172.21.15.57:0/4174726756 >> v1:172.21.15.179:6789/0 conn(0x7feb240af670 legacy=0x7feb2413e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:11.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.708+0000 7feb2a082640 1 -- 172.21.15.57:0/4174726756 shutdown_connections 2024-03-29T10:50:11.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.708+0000 7feb2a082640 1 -- 172.21.15.57:0/4174726756 >> 172.21.15.57:0/4174726756 conn(0x7feb24066a00 msgr2=0x7feb24140910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:11.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.708+0000 7feb2a082640 1 -- 172.21.15.57:0/4174726756 shutdown_connections 2024-03-29T10:50:11.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.708+0000 7feb2a082640 1 -- 172.21.15.57:0/4174726756 wait complete. 2024-03-29T10:50:11.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:11.708+0000 7feb2a082640 1 librados: shutdown 2024-03-29T10:50:11.705 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:18.511 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:18.684 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.702+0000 7f3d606f1640 10 monclient: get_monmap_and_config 2024-03-29T10:50:18.685 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.702+0000 7f3d606f1640 10 monclient: build_initial_monmap 2024-03-29T10:50:18.685 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.702+0000 7f3d606f1640 10 monclient: monmap: 2024-03-29T10:50:18.685 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:50:18.685 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:18.685 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:50:18.703875+0000 2024-03-29T10:50:18.685 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:50:18.703875+0000 2024-03-29T10:50:18.686 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:18.686 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:18.686 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:50:18.686 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:50:18.686 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:50:18.686 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:18.686 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.703+0000 7f3d606f1640 1 Processor -- start 2024-03-29T10:50:18.686 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.703+0000 7f3d606f1640 1 -- start start 2024-03-29T10:50:18.687 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.703+0000 7f3d606f1640 10 monclient: init 2024-03-29T10:50:18.688 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.705+0000 7f3d606f1640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:18.688 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.705+0000 7f3d606f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:50:18.688 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.705+0000 7f3d606f1640 10 monclient(hunting): picked mon.noname-b con 0x7f3d580b4920 addr v1:172.21.15.57:6790/0 2024-03-29T10:50:18.688 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.705+0000 7f3d606f1640 10 monclient(hunting): picked mon.noname-c con 0x7f3d580b1790 addr v1:172.21.15.179:6789/0 2024-03-29T10:50:18.688 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.705+0000 7f3d5e48d640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7f3d580b4920 0x7f3d5806e510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:57984/0 (socket says 172.21.15.57:57984) 2024-03-29T10:50:18.688 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.705+0000 7f3d606f1640 10 monclient(hunting): picked mon.noname-a con 0x7f3d58146790 addr v1:172.21.15.57:6789/0 2024-03-29T10:50:18.688 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.705+0000 7f3d5e48d640 1 -- 172.21.15.57:0/569549922 learned_addr learned my addr 172.21.15.57:0/569549922 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:50:18.688 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.705+0000 7f3d606f1640 1 -- 172.21.15.57:0/569549922 --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d580b4070 con 0x7f3d58146790 2024-03-29T10:50:18.688 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.705+0000 7f3d606f1640 1 -- 172.21.15.57:0/569549922 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5806fb80 con 0x7f3d580b1790 2024-03-29T10:50:18.688 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.705+0000 7f3d606f1640 1 -- 172.21.15.57:0/569549922 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d580b42c0 con 0x7f3d580b4920 2024-03-29T10:50:18.688 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.705+0000 7f3d606f1640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:18.688 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.705+0000 7f3d606f1640 10 monclient(hunting): authenticate will time out at 1570.882632s 2024-03-29T10:50:18.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.706+0000 7f3d5d48b640 1 -- 172.21.15.57:0/569549922 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171421188 0 0) 0x7f3d5806fb80 con 0x7f3d580b1790 2024-03-29T10:50:18.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.706+0000 7f3d5d48b640 10 monclient(hunting): _negotiate 2024-03-29T10:50:18.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.706+0000 7f3d5d48b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:18.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.706+0000 7f3d5d48b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:18.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.706+0000 7f3d5d48b640 10 monclient(hunting): my global_id is 4588 2024-03-29T10:50:18.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.706+0000 7f3d5d48b640 1 -- 172.21.15.57:0/569549922 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d48003520 con 0x7f3d580b1790 2024-03-29T10:50:18.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.706+0000 7f3d5d48b640 1 -- 172.21.15.57:0/569549922 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098574904 0 0) 0x7f3d580b42c0 con 0x7f3d580b4920 2024-03-29T10:50:18.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.706+0000 7f3d5d48b640 10 monclient(hunting): _negotiate 2024-03-29T10:50:18.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.706+0000 7f3d5d48b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:18.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.706+0000 7f3d5d48b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:18.690 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.706+0000 7f3d5d48b640 10 monclient(hunting): my global_id is 4652 2024-03-29T10:50:18.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.706+0000 7f3d5d48b640 1 -- 172.21.15.57:0/569549922 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d480048a0 con 0x7f3d580b4920 2024-03-29T10:50:18.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.706+0000 7f3d5d48b640 1 -- 172.21.15.57:0/569549922 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 229458662 0 0) 0x7f3d48003520 con 0x7f3d580b1790 2024-03-29T10:50:18.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.706+0000 7f3d5d48b640 1 -- 172.21.15.57:0/569549922 >> v1:172.21.15.57:6790/0 conn(0x7f3d580b4920 legacy=0x7f3d5806e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:18.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.706+0000 7f3d5d48b640 1 -- 172.21.15.57:0/569549922 >> v1:172.21.15.57:6789/0 conn(0x7f3d58146790 legacy=0x7f3d58146c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:18.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.706+0000 7f3d5d48b640 10 monclient: _finish_hunting 0 2024-03-29T10:50:18.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.706+0000 7f3d5d48b640 1 monclient: found mon.noname-c 2024-03-29T10:50:18.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.706+0000 7f3d5d48b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:18.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.706+0000 7f3d5d48b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.179:6789/0 2024-03-29T10:50:18.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.706+0000 7f3d5d48b640 1 -- 172.21.15.57:0/569549922 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d580b22c0 con 0x7f3d580b1790 2024-03-29T10:50:18.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.707+0000 7f3d5d48b640 10 monclient: _finish_auth 0 2024-03-29T10:50:18.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.707+0000 7f3d5d48b640 10 monclient: _check_auth_tickets 2024-03-29T10:50:18.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.707+0000 7f3d5d48b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:18.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.707+0000 7f3d5d48b640 1 -- 172.21.15.57:0/569549922 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f3d44001670 con 0x7f3d580b1790 2024-03-29T10:50:18.691 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.707+0000 7f3d5d48b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:18.692 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.707+0000 7f3d5d48b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:18.692 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.707+0000 7f3d5d48b640 10 monclient: dump: 2024-03-29T10:50:18.692 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:50:18.692 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:18.692 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:18.692 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:18.692 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:18.692 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:18.693 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:50:18.693 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:50:18.693 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:50:18.693 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:18.693 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.707+0000 7f3d606f1640 5 monclient: authenticate success, global_id 4588 2024-03-29T10:50:18.693 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.707+0000 7f3d606f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:18.693 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.707+0000 7f3d5d48b640 1 -- 172.21.15.57:0/569549922 <== mon.1 v1:172.21.15.179:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d440017d0 con 0x7f3d580b1790 2024-03-29T10:50:18.693 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.707+0000 7f3d5d48b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:18.693 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.707+0000 7f3d5d48b640 1 -- 172.21.15.57:0/569549922 <== mon.1 v1:172.21.15.179:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f3d44001be0 con 0x7f3d580b1790 2024-03-29T10:50:18.693 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.707+0000 7f3d5d48b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:18.693 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.707+0000 7f3d5d48b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:18.693 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.707+0000 7f3d5d48b640 10 monclient: dump: 2024-03-29T10:50:18.693 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:50:18.693 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:18.693 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:18.694 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:18.694 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:18.694 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:18.694 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:50:18.694 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:50:18.694 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:50:18.694 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:18.694 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.707+0000 7f3d606f1640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:18.694 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.707+0000 7f3d606f1640 4 set_mon_vals no callback set 2024-03-29T10:50:18.696 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.709+0000 7f3d606f1640 10 monclient: shutdown 2024-03-29T10:50:18.696 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.709+0000 7f3d606f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:18.696 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.709+0000 7f3d606f1640 1 -- 172.21.15.57:0/569549922 >> v1:172.21.15.179:6789/0 conn(0x7f3d580b1790 legacy=0x7f3d580b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:18.697 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.710+0000 7f3d606f1640 1 -- 172.21.15.57:0/569549922 shutdown_connections 2024-03-29T10:50:18.697 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.710+0000 7f3d606f1640 1 -- 172.21.15.57:0/569549922 >> 172.21.15.57:0/569549922 conn(0x7f3d58066a00 msgr2=0x7f3d58066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:18.697 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.710+0000 7f3d606f1640 1 -- 172.21.15.57:0/569549922 shutdown_connections 2024-03-29T10:50:18.697 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.710+0000 7f3d606f1640 1 -- 172.21.15.57:0/569549922 wait complete. 2024-03-29T10:50:18.697 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.711+0000 7f3d606f1640 10 monclient: build_initial_monmap 2024-03-29T10:50:18.697 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.711+0000 7f3d606f1640 10 monclient: monmap: 2024-03-29T10:50:18.697 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:50:18.697 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:18.697 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:50:18.697 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:50:18.697 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:18.697 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:18.697 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:50:18.697 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:50:18.697 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:50:18.698 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:18.698 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.711+0000 7f3d606f1640 1 librados: starting msgr at 2024-03-29T10:50:18.698 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.711+0000 7f3d606f1640 1 librados: starting objecter 2024-03-29T10:50:18.698 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.711+0000 7f3d606f1640 1 Processor -- start 2024-03-29T10:50:18.698 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.711+0000 7f3d606f1640 1 -- start start 2024-03-29T10:50:18.698 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.711+0000 7f3d606f1640 1 librados: setting wanted keys 2024-03-29T10:50:18.698 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.711+0000 7f3d606f1640 1 librados: calling monclient init 2024-03-29T10:50:18.698 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.711+0000 7f3d606f1640 10 monclient: init 2024-03-29T10:50:18.698 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.712+0000 7f3d606f1640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:18.698 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.712+0000 7f3d606f1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:50:18.698 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.712+0000 7f3d606f1640 10 monclient(hunting): picked mon.noname-a con 0x7f3d580b1790 addr v1:172.21.15.57:6789/0 2024-03-29T10:50:18.698 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.712+0000 7f3d606f1640 10 monclient(hunting): picked mon.noname-b con 0x7f3d580b4920 addr v1:172.21.15.179:6789/0 2024-03-29T10:50:18.698 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.712+0000 7f3d5e48d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f3d580b1790 0x7f3d58131340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:37614/0 (socket says 172.21.15.57:37614) 2024-03-29T10:50:18.698 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.713+0000 7f3d5e48d640 1 -- 172.21.15.57:0/3833975821 learned_addr learned my addr 172.21.15.57:0/3833975821 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:50:18.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.713+0000 7f3d606f1640 10 monclient(hunting): picked mon.noname-c con 0x7f3d58146790 addr v1:172.21.15.57:6790/0 2024-03-29T10:50:18.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.713+0000 7f3d606f1640 1 -- 172.21.15.57:0/3833975821 --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5813ce70 con 0x7f3d580b1790 2024-03-29T10:50:18.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.713+0000 7f3d606f1640 1 -- 172.21.15.57:0/3833975821 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d58152500 con 0x7f3d580b4920 2024-03-29T10:50:18.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.713+0000 7f3d56ffd640 10 client.?.objecter ms_handle_connect 0x7f3d580b1790 2024-03-29T10:50:18.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.713+0000 7f3d606f1640 1 -- 172.21.15.57:0/3833975821 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5813df20 con 0x7f3d58146790 2024-03-29T10:50:18.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.713+0000 7f3d56ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:18.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.713+0000 7f3d56ffd640 10 client.?.objecter ms_handle_connect 0x7f3d580b4920 2024-03-29T10:50:18.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.713+0000 7f3d56ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:18.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.713+0000 7f3d606f1640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:18.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.713+0000 7f3d606f1640 10 monclient(hunting): authenticate will time out at 1570.890533s 2024-03-29T10:50:18.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.713+0000 7f3d56ffd640 10 client.?.objecter ms_handle_connect 0x7f3d58146790 2024-03-29T10:50:18.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.713+0000 7f3d56ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:18.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.713+0000 7f3d56ffd640 1 -- 172.21.15.57:0/3833975821 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035003199 0 0) 0x7f3d58152500 con 0x7f3d580b4920 2024-03-29T10:50:18.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.713+0000 7f3d56ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:18.699 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.713+0000 7f3d56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:18.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.713+0000 7f3d56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:18.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.713+0000 7f3d56ffd640 10 monclient(hunting): my global_id is 4591 2024-03-29T10:50:18.700 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.713+0000 7f3d56ffd640 1 -- 172.21.15.57:0/3833975821 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d3c004eb0 con 0x7f3d580b4920 2024-03-29T10:50:18.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.714+0000 7f3d56ffd640 1 -- 172.21.15.57:0/3833975821 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439407505 0 0) 0x7f3d5813ce70 con 0x7f3d580b1790 2024-03-29T10:50:18.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.714+0000 7f3d56ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:18.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.714+0000 7f3d56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:18.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.714+0000 7f3d56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:18.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.714+0000 7f3d56ffd640 10 monclient(hunting): my global_id is 4587 2024-03-29T10:50:18.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.714+0000 7f3d56ffd640 1 -- 172.21.15.57:0/3833975821 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d58152500 con 0x7f3d580b1790 2024-03-29T10:50:18.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.714+0000 7f3d56ffd640 1 -- 172.21.15.57:0/3833975821 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3481208015 0 0) 0x7f3d3c004eb0 con 0x7f3d580b4920 2024-03-29T10:50:18.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.714+0000 7f3d56ffd640 1 -- 172.21.15.57:0/3833975821 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d5813ce70 con 0x7f3d580b4920 2024-03-29T10:50:18.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.714+0000 7f3d56ffd640 1 -- 172.21.15.57:0/3833975821 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787837741 0 0) 0x7f3d5813df20 con 0x7f3d58146790 2024-03-29T10:50:18.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.714+0000 7f3d56ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:50:18.702 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.714+0000 7f3d56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:18.703 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.714+0000 7f3d56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:18.703 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.714+0000 7f3d56ffd640 10 monclient(hunting): my global_id is 4655 2024-03-29T10:50:18.703 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.714+0000 7f3d56ffd640 1 -- 172.21.15.57:0/3833975821 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d3c004eb0 con 0x7f3d58146790 2024-03-29T10:50:18.703 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d56ffd640 1 -- 172.21.15.57:0/3833975821 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f3d44001be0 con 0x7f3d580b4920 2024-03-29T10:50:18.703 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d56ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:18.703 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d56ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:18.703 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d56ffd640 10 monclient(hunting): dump: 2024-03-29T10:50:18.703 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:50:18.703 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:18.703 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:18.703 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:18.703 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:18.704 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:18.704 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:50:18.704 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:50:18.704 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:50:18.704 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:18.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d56ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:18.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d56ffd640 1 -- 172.21.15.57:0/3833975821 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 61503837 0 0) 0x7f3d3c004eb0 con 0x7f3d58146790 2024-03-29T10:50:18.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d56ffd640 1 -- 172.21.15.57:0/3833975821 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d5813df20 con 0x7f3d58146790 2024-03-29T10:50:18.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d56ffd640 1 -- 172.21.15.57:0/3833975821 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f3d500016d0 con 0x7f3d58146790 2024-03-29T10:50:18.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d56ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:18.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d56ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:18.704 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d56ffd640 10 monclient(hunting): dump: 2024-03-29T10:50:18.704 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:50:18.705 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:18.705 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:18.705 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:18.705 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:18.705 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:18.705 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:50:18.705 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:50:18.705 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:50:18.705 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:18.705 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d56ffd640 1 -- 172.21.15.57:0/3833975821 <== mon.1 v1:172.21.15.179:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 873366724 0 0) 0x7f3d5813ce70 con 0x7f3d580b4920 2024-03-29T10:50:18.705 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d56ffd640 1 -- 172.21.15.57:0/3833975821 >> v1:172.21.15.57:6790/0 conn(0x7f3d58146790 legacy=0x7f3d58151df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:18.705 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d56ffd640 1 -- 172.21.15.57:0/3833975821 >> v1:172.21.15.57:6789/0 conn(0x7f3d580b1790 legacy=0x7f3d58131340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:18.705 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d56ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:50:18.705 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d56ffd640 1 monclient: found mon.b 2024-03-29T10:50:18.705 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:18.706 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d56ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:50:18.706 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d56ffd640 1 -- 172.21.15.57:0/3833975821 --> v1:172.21.15.179:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d58135c00 con 0x7f3d580b4920 2024-03-29T10:50:18.706 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d56ffd640 10 monclient: _finish_auth 0 2024-03-29T10:50:18.706 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d56ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:50:18.706 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:18.706 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d606f1640 5 monclient: authenticate success, global_id 4591 2024-03-29T10:50:18.706 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d606f1640 10 monclient: _renew_subs 2024-03-29T10:50:18.706 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d606f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:50:18.706 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.715+0000 7f3d606f1640 1 -- 172.21.15.57:0/3833975821 --> v1:172.21.15.179:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d58152880 con 0x7f3d580b4920 2024-03-29T10:50:18.706 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.716+0000 7f3d606f1640 10 client.4591.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:18.706 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.716+0000 7f3d606f1640 10 monclient: _renew_subs 2024-03-29T10:50:18.706 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.716+0000 7f3d606f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:50:18.706 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.716+0000 7f3d606f1640 1 -- 172.21.15.57:0/3833975821 --> v1:172.21.15.179:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d5813c890 con 0x7f3d580b4920 2024-03-29T10:50:18.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.716+0000 7f3d606f1640 1 librados: init done 2024-03-29T10:50:18.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.716+0000 7f3d56ffd640 1 -- 172.21.15.57:0/3833975821 <== mon.1 v1:172.21.15.179:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d44000ed0 con 0x7f3d580b4920 2024-03-29T10:50:18.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.716+0000 7f3d56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:18.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.716+0000 7f3d56ffd640 1 -- 172.21.15.57:0/3833975821 <== mon.1 v1:172.21.15.179:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7f3d44003a40 con 0x7f3d580b4920 2024-03-29T10:50:18.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.716+0000 7f3d56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:18.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.716+0000 7f3d56ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:18.707 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.716+0000 7f3d56ffd640 10 monclient: dump: 2024-03-29T10:50:18.707 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:50:18.707 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:18.707 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:18.707 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:18.707 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:18.707 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:18.707 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:50:18.708 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:50:18.708 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:50:18.708 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:18.708 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.716+0000 7f3d5cc8a640 4 set_mon_vals no callback set 2024-03-29T10:50:18.708 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.717+0000 7f3d606f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:18.708 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.717+0000 7f3d606f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:18.708 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.717+0000 7f3d606f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:50:18.708 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.717+0000 7f3d606f1640 1 -- 172.21.15.57:0/3833975821 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d40005080 con 0x7f3d580b4920 2024-03-29T10:50:18.708 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.718+0000 7f3d56ffd640 1 -- 172.21.15.57:0/3833975821 <== mon.1 v1:172.21.15.179:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7f3d44019a10 con 0x7f3d580b4920 2024-03-29T10:50:18.708 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.719+0000 7f3d56ffd640 1 -- 172.21.15.57:0/3833975821 <== mon.1 v1:172.21.15.179:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3486612304 0 0) 0x7f3d44083890 con 0x7f3d580b4920 2024-03-29T10:50:18.708 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.719+0000 7f3d56ffd640 10 client.4591.objecter ms_dispatch 0x7f3d58000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:50:18.708 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.719+0000 7f3d56ffd640 3 client.4591.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:50:18.708 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.719+0000 7f3d56ffd640 3 client.4591.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:50:18.708 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.719+0000 7f3d56ffd640 20 client.4591.objecter dump_active .. 0 homeless 2024-03-29T10:50:18.708 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.719+0000 7f3d5e48d640 10 monclient: get_auth_request con 0x7f3d3c06dfe0 auth_method 0 2024-03-29T10:50:18.709 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.719+0000 7f3d56ffd640 10 client.4591.objecter ms_handle_connect 0x7f3d3c06dfe0 2024-03-29T10:50:18.709 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.725+0000 7f3d56ffd640 1 -- 172.21.15.57:0/3833975821 <== mon.1 v1:172.21.15.179:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3d4404f170 con 0x7f3d580b4920 2024-03-29T10:50:18.709 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.725+0000 7f3d56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:18.709 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:18.725+0000 7f3d56ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:19.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:19.110+0000 7f3d606f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:19.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:19.110+0000 7f3d606f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:19.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:19.110+0000 7f3d606f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.179:6789/0 2024-03-29T10:50:19.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:19.110+0000 7f3d606f1640 1 -- 172.21.15.57:0/3833975821 --> v1:172.21.15.179:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d40005330 con 0x7f3d580b4920 2024-03-29T10:50:19.095 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:19.111+0000 7f3d56ffd640 1 -- 172.21.15.57:0/3833975821 <== mon.1 v1:172.21.15.179:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2357246973) 0x7f3d440510a0 con 0x7f3d580b4920 2024-03-29T10:50:19.095 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:19.112+0000 7f3d56ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:19.095 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:19.112+0000 7f3d56ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:19.096 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:50:19.096 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":11,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:49:40.047000+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:49:40.047000+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:19.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:19.115+0000 7f3d606f1640 10 librados: watch_flush enter 2024-03-29T10:50:19.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:19.115+0000 7f3d606f1640 10 librados: watch_flush exit 2024-03-29T10:50:19.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:19.115+0000 7f3d606f1640 20 client.4591.objecter shutdown clearing up homeless session... 2024-03-29T10:50:19.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:19.115+0000 7f3d606f1640 10 client.4591.objecter successfully canceled tick 2024-03-29T10:50:19.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:19.115+0000 7f3d606f1640 1 -- 172.21.15.57:0/3833975821 >> v1:172.21.15.179:6816/2698295936 conn(0x7f3d3c06dfe0 legacy=0x7f3d3c0704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:19.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:19.115+0000 7f3d606f1640 10 monclient: shutdown 2024-03-29T10:50:19.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:19.115+0000 7f3d606f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:19.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:19.115+0000 7f3d606f1640 1 -- 172.21.15.57:0/3833975821 >> v1:172.21.15.179:6789/0 conn(0x7f3d580b4920 legacy=0x7f3d581516e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:19.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:19.115+0000 7f3d606f1640 1 -- 172.21.15.57:0/3833975821 shutdown_connections 2024-03-29T10:50:19.100 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:19.115+0000 7f3d606f1640 1 -- 172.21.15.57:0/3833975821 >> 172.21.15.57:0/3833975821 conn(0x7f3d58066a00 msgr2=0x7f3d5806e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:19.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:19.116+0000 7f3d606f1640 1 -- 172.21.15.57:0/3833975821 shutdown_connections 2024-03-29T10:50:19.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:19.116+0000 7f3d606f1640 1 -- 172.21.15.57:0/3833975821 wait complete. 2024-03-29T10:50:19.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:19.116+0000 7f3d606f1640 1 librados: shutdown 2024-03-29T10:50:19.112 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:25.919 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:26.074 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.091+0000 7fdd1dae0640 10 monclient: get_monmap_and_config 2024-03-29T10:50:26.074 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.091+0000 7fdd1dae0640 10 monclient: build_initial_monmap 2024-03-29T10:50:26.074 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.091+0000 7fdd1dae0640 10 monclient: monmap: 2024-03-29T10:50:26.074 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:50:26.074 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:26.074 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:50:26.092997+0000 2024-03-29T10:50:26.074 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:50:26.092997+0000 2024-03-29T10:50:26.074 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:26.074 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:26.075 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:50:26.075 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:50:26.075 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:50:26.075 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:26.075 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.092+0000 7fdd1dae0640 1 Processor -- start 2024-03-29T10:50:26.075 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.092+0000 7fdd1dae0640 1 -- start start 2024-03-29T10:50:26.075 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.092+0000 7fdd1dae0640 10 monclient: init 2024-03-29T10:50:26.078 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.093+0000 7fdd1dae0640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:26.078 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.093+0000 7fdd1dae0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:50:26.078 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.093+0000 7fdd1dae0640 10 monclient(hunting): picked mon.noname-c con 0x7fdd1806c250 addr v1:172.21.15.179:6789/0 2024-03-29T10:50:26.078 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.093+0000 7fdd1dae0640 10 monclient(hunting): picked mon.noname-b con 0x7fdd18072400 addr v1:172.21.15.57:6790/0 2024-03-29T10:50:26.078 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.093+0000 7fdd1dae0640 10 monclient(hunting): picked mon.noname-a con 0x7fdd1806c630 addr v1:172.21.15.57:6789/0 2024-03-29T10:50:26.078 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.093+0000 7fdd1dae0640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1806d760 con 0x7fdd1806c630 2024-03-29T10:50:26.078 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.093+0000 7fdd1dae0640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd18070500 con 0x7fdd1806c250 2024-03-29T10:50:26.078 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.093+0000 7fdd1dae0640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd180aff50 con 0x7fdd18072400 2024-03-29T10:50:26.078 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.093+0000 7fdd1dae0640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:26.078 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.093+0000 7fdd1dae0640 10 monclient(hunting): authenticate will time out at 1578.270409s 2024-03-29T10:50:26.079 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.095+0000 7fdd17fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fdd1806c630 0x7fdd18151ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:38672/0 (socket says 172.21.15.57:38672) 2024-03-29T10:50:26.079 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.095+0000 7fdd17fff640 1 -- 172.21.15.57:0/1557911866 learned_addr learned my addr 172.21.15.57:0/1557911866 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:50:26.080 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.096+0000 7fdd167fc640 1 -- 172.21.15.57:0/1557911866 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335417127 0 0) 0x7fdd1806d760 con 0x7fdd1806c630 2024-03-29T10:50:26.080 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.096+0000 7fdd167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:26.080 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.096+0000 7fdd167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:26.080 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.096+0000 7fdd167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:26.080 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.096+0000 7fdd167fc640 10 monclient(hunting): my global_id is 4590 2024-03-29T10:50:26.081 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.096+0000 7fdd167fc640 1 -- 172.21.15.57:0/1557911866 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd00003520 con 0x7fdd1806c630 2024-03-29T10:50:26.081 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.096+0000 7fdd167fc640 1 -- 172.21.15.57:0/1557911866 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3815060408 0 0) 0x7fdd18070500 con 0x7fdd1806c250 2024-03-29T10:50:26.081 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.096+0000 7fdd167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:50:26.081 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.096+0000 7fdd167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:26.081 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.096+0000 7fdd167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:26.081 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.096+0000 7fdd167fc640 10 monclient(hunting): my global_id is 4594 2024-03-29T10:50:26.081 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.096+0000 7fdd167fc640 1 -- 172.21.15.57:0/1557911866 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd1806d760 con 0x7fdd1806c250 2024-03-29T10:50:26.081 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.096+0000 7fdd167fc640 1 -- 172.21.15.57:0/1557911866 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 191569140 0 0) 0x7fdd00003520 con 0x7fdd1806c630 2024-03-29T10:50:26.081 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.096+0000 7fdd167fc640 1 -- 172.21.15.57:0/1557911866 >> v1:172.21.15.57:6790/0 conn(0x7fdd18072400 legacy=0x7fdd181465b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:50:26.083 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.099+0000 7fdd167fc640 1 -- 172.21.15.57:0/1557911866 >> v1:172.21.15.179:6789/0 conn(0x7fdd1806c250 legacy=0x7fdd18071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:26.083 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.099+0000 7fdd167fc640 10 monclient: _finish_hunting 0 2024-03-29T10:50:26.083 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.099+0000 7fdd167fc640 1 monclient: found mon.noname-a 2024-03-29T10:50:26.083 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.099+0000 7fdd167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:26.083 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.099+0000 7fdd167fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.57:6789/0 2024-03-29T10:50:26.084 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.099+0000 7fdd167fc640 1 -- 172.21.15.57:0/1557911866 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd180b00e0 con 0x7fdd1806c630 2024-03-29T10:50:26.084 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.099+0000 7fdd167fc640 10 monclient: _finish_auth 0 2024-03-29T10:50:26.084 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.099+0000 7fdd167fc640 10 monclient: _check_auth_tickets 2024-03-29T10:50:26.084 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.099+0000 7fdd167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:26.084 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.099+0000 7fdd167fc640 1 -- 172.21.15.57:0/1557911866 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fdd0c001610 con 0x7fdd1806c630 2024-03-29T10:50:26.084 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.099+0000 7fdd167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:26.084 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.099+0000 7fdd167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:26.084 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.099+0000 7fdd167fc640 10 monclient: dump: 2024-03-29T10:50:26.084 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:50:26.084 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:26.084 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:26.084 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:26.084 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:26.084 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:26.085 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:50:26.085 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:50:26.085 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:50:26.085 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:26.085 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.099+0000 7fdd1dae0640 5 monclient: authenticate success, global_id 4590 2024-03-29T10:50:26.085 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.099+0000 7fdd1dae0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:26.085 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.099+0000 7fdd167fc640 1 -- 172.21.15.57:0/1557911866 <== mon.0 v1:172.21.15.57:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd0c001830 con 0x7fdd1806c630 2024-03-29T10:50:26.085 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.100+0000 7fdd167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:26.085 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.100+0000 7fdd167fc640 1 -- 172.21.15.57:0/1557911866 <== mon.0 v1:172.21.15.57:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fdd0c001bc0 con 0x7fdd1806c630 2024-03-29T10:50:26.085 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.100+0000 7fdd167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:26.085 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.100+0000 7fdd167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:26.085 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.100+0000 7fdd167fc640 10 monclient: dump: 2024-03-29T10:50:26.085 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:50:26.085 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:26.085 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:26.086 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:26.086 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:26.086 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:26.086 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:50:26.086 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:50:26.086 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:50:26.086 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:26.086 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.100+0000 7fdd1dae0640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:26.086 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.100+0000 7fdd1dae0640 4 set_mon_vals no callback set 2024-03-29T10:50:26.086 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.100+0000 7fdd1dae0640 10 monclient: shutdown 2024-03-29T10:50:26.086 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.100+0000 7fdd1dae0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:26.086 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.100+0000 7fdd1dae0640 1 -- 172.21.15.57:0/1557911866 >> v1:172.21.15.57:6789/0 conn(0x7fdd1806c630 legacy=0x7fdd18151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:26.086 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.101+0000 7fdd1dae0640 1 -- 172.21.15.57:0/1557911866 shutdown_connections 2024-03-29T10:50:26.086 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.101+0000 7fdd1dae0640 1 -- 172.21.15.57:0/1557911866 >> 172.21.15.57:0/1557911866 conn(0x7fdd18067610 msgr2=0x7fdd180b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:26.087 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.101+0000 7fdd1dae0640 1 -- 172.21.15.57:0/1557911866 shutdown_connections 2024-03-29T10:50:26.088 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.101+0000 7fdd1dae0640 1 -- 172.21.15.57:0/1557911866 wait complete. 2024-03-29T10:50:26.088 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.102+0000 7fdd1dae0640 10 monclient: build_initial_monmap 2024-03-29T10:50:26.088 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.102+0000 7fdd1dae0640 10 monclient: monmap: 2024-03-29T10:50:26.088 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:50:26.088 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:26.088 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:50:26.088 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:50:26.088 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:26.089 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:26.089 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:50:26.089 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:50:26.089 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:50:26.089 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:26.089 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.102+0000 7fdd1dae0640 1 librados: starting msgr at 2024-03-29T10:50:26.089 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.102+0000 7fdd1dae0640 1 librados: starting objecter 2024-03-29T10:50:26.089 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.102+0000 7fdd1dae0640 1 Processor -- start 2024-03-29T10:50:26.089 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.103+0000 7fdd1dae0640 1 -- start start 2024-03-29T10:50:26.089 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.103+0000 7fdd1dae0640 1 librados: setting wanted keys 2024-03-29T10:50:26.089 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.103+0000 7fdd1dae0640 1 librados: calling monclient init 2024-03-29T10:50:26.089 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.103+0000 7fdd1dae0640 10 monclient: init 2024-03-29T10:50:26.089 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.104+0000 7fdd1dae0640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:26.089 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.104+0000 7fdd1dae0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:50:26.089 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.104+0000 7fdd1dae0640 10 monclient(hunting): picked mon.noname-b con 0x7fdd1806c250 addr v1:172.21.15.179:6789/0 2024-03-29T10:50:26.092 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.107+0000 7fdd16ffd640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7fdd1806c630 0x7fdd181e4c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:60007/0 (socket says 172.21.15.57:60007) 2024-03-29T10:50:26.092 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.107+0000 7fdd1dae0640 10 monclient(hunting): picked mon.noname-c con 0x7fdd1806c630 addr v1:172.21.15.57:6790/0 2024-03-29T10:50:26.092 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.108+0000 7fdd1dae0640 10 monclient(hunting): picked mon.noname-a con 0x7fdd18072400 addr v1:172.21.15.57:6789/0 2024-03-29T10:50:26.092 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.108+0000 7fdd16ffd640 1 -- 172.21.15.57:0/2418744330 learned_addr learned my addr 172.21.15.57:0/2418744330 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:50:26.092 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.108+0000 7fdd1dae0640 1 -- 172.21.15.57:0/2418744330 --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd180ae110 con 0x7fdd18072400 2024-03-29T10:50:26.092 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.108+0000 7fdd1dae0640 1 -- 172.21.15.57:0/2418744330 --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd180aed50 con 0x7fdd1806c250 2024-03-29T10:50:26.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.108+0000 7fdd1dae0640 1 -- 172.21.15.57:0/2418744330 --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd180af030 con 0x7fdd1806c630 2024-03-29T10:50:26.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.108+0000 7fdd1dae0640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:26.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.108+0000 7fdd1dae0640 10 monclient(hunting): authenticate will time out at 1578.285655s 2024-03-29T10:50:26.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.108+0000 7fdcfffff640 10 client.?.objecter ms_handle_connect 0x7fdd1806c630 2024-03-29T10:50:26.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.108+0000 7fdcfffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:26.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.108+0000 7fdcfffff640 10 client.?.objecter ms_handle_connect 0x7fdd18072400 2024-03-29T10:50:26.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.108+0000 7fdcfffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:26.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.109+0000 7fdcfffff640 1 -- 172.21.15.57:0/2418744330 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873876552 0 0) 0x7fdd180ae110 con 0x7fdd18072400 2024-03-29T10:50:26.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.109+0000 7fdcfffff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:26.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.109+0000 7fdcfffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:26.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.109+0000 7fdcfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:26.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.109+0000 7fdcfffff640 10 monclient(hunting): my global_id is 4593 2024-03-29T10:50:26.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.109+0000 7fdcfffff640 1 -- 172.21.15.57:0/2418744330 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd04004eb0 con 0x7fdd18072400 2024-03-29T10:50:26.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.109+0000 7fdcfffff640 1 -- 172.21.15.57:0/2418744330 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1693419193 0 0) 0x7fdd180af030 con 0x7fdd1806c630 2024-03-29T10:50:26.093 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.109+0000 7fdcfffff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:26.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.109+0000 7fdcfffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:26.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.109+0000 7fdcfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:26.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.109+0000 7fdcfffff640 10 monclient(hunting): my global_id is 4658 2024-03-29T10:50:26.094 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.109+0000 7fdcfffff640 1 -- 172.21.15.57:0/2418744330 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd180ae110 con 0x7fdd1806c630 2024-03-29T10:50:26.096 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.110+0000 7fdcfffff640 1 -- 172.21.15.57:0/2418744330 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3551334023 0 0) 0x7fdd180ae110 con 0x7fdd1806c630 2024-03-29T10:50:26.096 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.111+0000 7fdcfffff640 1 -- 172.21.15.57:0/2418744330 --> v1:172.21.15.57:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd180af030 con 0x7fdd1806c630 2024-03-29T10:50:26.096 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.111+0000 7fdcfffff640 1 -- 172.21.15.57:0/2418744330 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fdd08001710 con 0x7fdd1806c630 2024-03-29T10:50:26.096 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.111+0000 7fdcfffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:26.096 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.111+0000 7fdcfffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:50:26.096 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.111+0000 7fdcfffff640 10 monclient(hunting): dump: 2024-03-29T10:50:26.096 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:50:26.096 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:26.097 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:26.097 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:26.097 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:26.097 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:26.097 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:50:26.097 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:50:26.097 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:50:26.097 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:26.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.111+0000 7fdcfffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:26.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.111+0000 7fdcfffff640 10 client.?.objecter ms_handle_connect 0x7fdd1806c250 2024-03-29T10:50:26.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.111+0000 7fdcfffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:26.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.111+0000 7fdcfffff640 1 -- 172.21.15.57:0/2418744330 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2712386041 0 0) 0x7fdd04004eb0 con 0x7fdd18072400 2024-03-29T10:50:26.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.111+0000 7fdcfffff640 1 -- 172.21.15.57:0/2418744330 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd180ae110 con 0x7fdd18072400 2024-03-29T10:50:26.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.111+0000 7fdcfffff640 1 -- 172.21.15.57:0/2418744330 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486655202 0 0) 0x7fdd180aed50 con 0x7fdd1806c250 2024-03-29T10:50:26.097 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.111+0000 7fdcfffff640 10 monclient(hunting): _negotiate 2024-03-29T10:50:26.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.111+0000 7fdcfffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:26.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.112+0000 7fdcfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:26.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.112+0000 7fdcfffff640 10 monclient(hunting): my global_id is 4597 2024-03-29T10:50:26.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.112+0000 7fdcfffff640 1 -- 172.21.15.57:0/2418744330 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd04004eb0 con 0x7fdd1806c250 2024-03-29T10:50:26.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.112+0000 7fdcfffff640 1 -- 172.21.15.57:0/2418744330 <== mon.2 v1:172.21.15.57:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4024455521 0 0) 0x7fdd180af030 con 0x7fdd1806c630 2024-03-29T10:50:26.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.112+0000 7fdcfffff640 1 -- 172.21.15.57:0/2418744330 >> v1:172.21.15.179:6789/0 conn(0x7fdd1806c250 legacy=0x7fdd181e4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:26.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.112+0000 7fdcfffff640 1 -- 172.21.15.57:0/2418744330 >> v1:172.21.15.57:6789/0 conn(0x7fdd18072400 legacy=0x7fdd181eeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:26.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.112+0000 7fdcfffff640 10 monclient: _finish_hunting 0 2024-03-29T10:50:26.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.112+0000 7fdcfffff640 1 monclient: found mon.c 2024-03-29T10:50:26.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.112+0000 7fdcfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:26.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.112+0000 7fdcfffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:50:26.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.112+0000 7fdcfffff640 1 -- 172.21.15.57:0/2418744330 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd180af310 con 0x7fdd1806c630 2024-03-29T10:50:26.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.112+0000 7fdcfffff640 10 monclient: _finish_auth 0 2024-03-29T10:50:26.098 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.112+0000 7fdcfffff640 10 monclient: _check_auth_tickets 2024-03-29T10:50:26.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.112+0000 7fdcfffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:26.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.112+0000 7fdcfffff640 1 -- 172.21.15.57:0/2418744330 <== mon.2 v1:172.21.15.57:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd08001870 con 0x7fdd1806c630 2024-03-29T10:50:26.099 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.112+0000 7fdcfffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:26.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.112+0000 7fdd15ffb640 4 set_mon_vals no callback set 2024-03-29T10:50:26.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.113+0000 7fdcfffff640 1 -- 172.21.15.57:0/2418744330 <== mon.2 v1:172.21.15.57:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fdd08004860 con 0x7fdd1806c630 2024-03-29T10:50:26.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.113+0000 7fdcfffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:26.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.113+0000 7fdcfffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:26.101 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.113+0000 7fdcfffff640 10 monclient: dump: 2024-03-29T10:50:26.101 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:50:26.101 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:26.101 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:26.101 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:26.102 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:26.102 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:26.102 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:50:26.102 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:50:26.102 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:50:26.102 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:26.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.113+0000 7fdd1dae0640 5 monclient: authenticate success, global_id 4658 2024-03-29T10:50:26.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.113+0000 7fdd1dae0640 10 monclient: _renew_subs 2024-03-29T10:50:26.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.113+0000 7fdd1dae0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:50:26.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.113+0000 7fdd1dae0640 1 -- 172.21.15.57:0/2418744330 --> v1:172.21.15.57:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd180adb10 con 0x7fdd1806c630 2024-03-29T10:50:26.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.113+0000 7fdd1dae0640 10 client.4658.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:26.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.113+0000 7fdd1dae0640 10 monclient: _renew_subs 2024-03-29T10:50:26.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.113+0000 7fdd1dae0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:50:26.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.113+0000 7fdd1dae0640 1 -- 172.21.15.57:0/2418744330 --> v1:172.21.15.57:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd180ad100 con 0x7fdd1806c630 2024-03-29T10:50:26.102 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.113+0000 7fdd1dae0640 1 librados: init done 2024-03-29T10:50:26.103 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.115+0000 7fdd1dae0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:26.106 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.115+0000 7fdd1dae0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:26.106 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.115+0000 7fdd1dae0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:50:26.106 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.115+0000 7fdd1dae0640 1 -- 172.21.15.57:0/2418744330 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd18075f10 con 0x7fdd1806c630 2024-03-29T10:50:26.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.117+0000 7fdcfffff640 1 -- 172.21.15.57:0/2418744330 <== mon.2 v1:172.21.15.57:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fdd0801a830 con 0x7fdd1806c630 2024-03-29T10:50:26.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.117+0000 7fdcfffff640 1 -- 172.21.15.57:0/2418744330 <== mon.2 v1:172.21.15.57:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3486612304 0 0) 0x7fdd080848a0 con 0x7fdd1806c630 2024-03-29T10:50:26.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.117+0000 7fdcfffff640 10 client.4658.objecter ms_dispatch 0x7fdd18000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:50:26.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.117+0000 7fdcfffff640 3 client.4658.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:50:26.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.117+0000 7fdcfffff640 3 client.4658.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:50:26.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.117+0000 7fdcfffff640 20 client.4658.objecter dump_active .. 0 homeless 2024-03-29T10:50:26.107 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.118+0000 7fdd177fe640 10 monclient: get_auth_request con 0x7fdd0406dec0 auth_method 0 2024-03-29T10:50:26.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.121+0000 7fdcfffff640 10 client.4658.objecter ms_handle_connect 0x7fdd0406dec0 2024-03-29T10:50:26.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.123+0000 7fdcfffff640 1 -- 172.21.15.57:0/2418744330 <== mon.2 v1:172.21.15.57:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdd0804ffb0 con 0x7fdd1806c630 2024-03-29T10:50:26.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.123+0000 7fdcfffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:26.108 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.123+0000 7fdcfffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:26.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.471+0000 7fdd1dae0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:26.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.471+0000 7fdd1dae0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:26.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.471+0000 7fdd1dae0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.57:6790/0 2024-03-29T10:50:26.455 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.471+0000 7fdd1dae0640 1 -- 172.21.15.57:0/2418744330 --> v1:172.21.15.57:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd18071a60 con 0x7fdd1806c630 2024-03-29T10:50:26.455 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.473+0000 7fdcfffff640 1 -- 172.21.15.57:0/2418744330 <== mon.2 v1:172.21.15.57:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2357246973) 0x7fdd080520b0 con 0x7fdd1806c630 2024-03-29T10:50:26.455 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.473+0000 7fdcfffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:26.455 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.473+0000 7fdcfffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:26.457 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:50:26.457 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":11,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:49:40.047000+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:49:40.047000+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:26.461 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.477+0000 7fdd1dae0640 10 librados: watch_flush enter 2024-03-29T10:50:26.461 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.478+0000 7fdd1dae0640 10 librados: watch_flush exit 2024-03-29T10:50:26.461 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.478+0000 7fdd1dae0640 20 client.4658.objecter shutdown clearing up homeless session... 2024-03-29T10:50:26.461 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.478+0000 7fdd1dae0640 10 client.4658.objecter successfully canceled tick 2024-03-29T10:50:26.462 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.480+0000 7fdd1dae0640 1 -- 172.21.15.57:0/2418744330 >> v1:172.21.15.179:6816/2698295936 conn(0x7fdd0406dec0 legacy=0x7fdd04070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:26.462 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.480+0000 7fdd1dae0640 10 monclient: shutdown 2024-03-29T10:50:26.462 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.480+0000 7fdd1dae0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:26.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.480+0000 7fdd1dae0640 1 -- 172.21.15.57:0/2418744330 >> v1:172.21.15.57:6790/0 conn(0x7fdd1806c630 legacy=0x7fdd181e4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:26.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.480+0000 7fdd1dae0640 1 -- 172.21.15.57:0/2418744330 shutdown_connections 2024-03-29T10:50:26.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.480+0000 7fdd1dae0640 1 -- 172.21.15.57:0/2418744330 >> 172.21.15.57:0/2418744330 conn(0x7fdd18067610 msgr2=0x7fdd18070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:26.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.480+0000 7fdd1dae0640 1 -- 172.21.15.57:0/2418744330 shutdown_connections 2024-03-29T10:50:26.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.480+0000 7fdd1dae0640 1 -- 172.21.15.57:0/2418744330 wait complete. 2024-03-29T10:50:26.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:26.481+0000 7fdd1dae0640 1 librados: shutdown 2024-03-29T10:50:26.481 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:33.288 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:50:33.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.454+0000 7fd1070ae640 10 monclient: get_monmap_and_config 2024-03-29T10:50:33.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.454+0000 7fd1070ae640 10 monclient: build_initial_monmap 2024-03-29T10:50:33.437 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.454+0000 7fd1070ae640 10 monclient: monmap: 2024-03-29T10:50:33.437 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:50:33.437 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:33.437 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:50:33.456386+0000 2024-03-29T10:50:33.437 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:50:33.456386+0000 2024-03-29T10:50:33.437 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:33.437 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:33.438 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:50:33.438 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-c 2024-03-29T10:50:33.438 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-b 2024-03-29T10:50:33.438 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:33.438 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.456+0000 7fd1070ae640 1 Processor -- start 2024-03-29T10:50:33.438 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.456+0000 7fd1070ae640 1 -- start start 2024-03-29T10:50:33.439 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.456+0000 7fd1070ae640 10 monclient: init 2024-03-29T10:50:33.439 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.457+0000 7fd1070ae640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:33.439 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.457+0000 7fd1070ae640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:50:33.439 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.457+0000 7fd1070ae640 10 monclient(hunting): picked mon.noname-b con 0x7fd1001457f0 addr v1:172.21.15.57:6790/0 2024-03-29T10:50:33.446 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.458+0000 7fd1070ae640 10 monclient(hunting): picked mon.noname-c con 0x7fd10013f650 addr v1:172.21.15.179:6789/0 2024-03-29T10:50:33.446 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.458+0000 7fd1070ae640 10 monclient(hunting): picked mon.noname-a con 0x7fd1000a9d70 addr v1:172.21.15.57:6789/0 2024-03-29T10:50:33.446 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.458+0000 7fd1070ae640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1000aaef0 con 0x7fd1000a9d70 2024-03-29T10:50:33.446 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.458+0000 7fd1070ae640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd100145bd0 con 0x7fd10013f650 2024-03-29T10:50:33.446 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.458+0000 7fd1070ae640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd100145eb0 con 0x7fd1001457f0 2024-03-29T10:50:33.446 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.458+0000 7fd1070ae640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:33.446 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.458+0000 7fd1070ae640 10 monclient(hunting): authenticate will time out at 1585.635448s 2024-03-29T10:50:33.446 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.458+0000 7fd10564b640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fd1000a9d70 0x7fd10014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:38032/0 (socket says 172.21.15.57:38032) 2024-03-29T10:50:33.446 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.458+0000 7fd10564b640 1 -- 172.21.15.57:0/62720603 learned_addr learned my addr 172.21.15.57:0/62720603 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:50:33.446 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.460+0000 7fd0ff7fe640 1 -- 172.21.15.57:0/62720603 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840427245 0 0) 0x7fd1000aaef0 con 0x7fd1000a9d70 2024-03-29T10:50:33.446 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.460+0000 7fd0ff7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:50:33.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.460+0000 7fd0ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:33.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.460+0000 7fd0ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:33.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.460+0000 7fd0ff7fe640 10 monclient(hunting): my global_id is 4596 2024-03-29T10:50:33.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.460+0000 7fd0ff7fe640 1 -- 172.21.15.57:0/62720603 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0e8003520 con 0x7fd1000a9d70 2024-03-29T10:50:33.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.460+0000 7fd0ff7fe640 1 -- 172.21.15.57:0/62720603 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901688370 0 0) 0x7fd100145eb0 con 0x7fd1001457f0 2024-03-29T10:50:33.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.460+0000 7fd0ff7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:50:33.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.460+0000 7fd0ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:33.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.460+0000 7fd0ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:33.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.460+0000 7fd0ff7fe640 10 monclient(hunting): my global_id is 4661 2024-03-29T10:50:33.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.460+0000 7fd0ff7fe640 1 -- 172.21.15.57:0/62720603 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1000aaef0 con 0x7fd1001457f0 2024-03-29T10:50:33.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.461+0000 7fd0ff7fe640 1 -- 172.21.15.57:0/62720603 <== mon.2 v1:172.21.15.57:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 986069047 0 0) 0x7fd1000aaef0 con 0x7fd1001457f0 2024-03-29T10:50:33.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.461+0000 7fd0ff7fe640 1 -- 172.21.15.57:0/62720603 >> v1:172.21.15.179:6789/0 conn(0x7fd10013f650 legacy=0x7fd1000b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:33.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.461+0000 7fd0ff7fe640 1 -- 172.21.15.57:0/62720603 >> v1:172.21.15.57:6789/0 conn(0x7fd1000a9d70 legacy=0x7fd10014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:33.447 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.461+0000 7fd0ff7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:50:33.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.461+0000 7fd0ff7fe640 1 monclient: found mon.noname-b 2024-03-29T10:50:33.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.461+0000 7fd0ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:33.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.461+0000 7fd0ff7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.57:6790/0 2024-03-29T10:50:33.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.461+0000 7fd0ff7fe640 1 -- 172.21.15.57:0/62720603 --> v1:172.21.15.57:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd100146340 con 0x7fd1001457f0 2024-03-29T10:50:33.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.461+0000 7fd0ff7fe640 10 monclient: _finish_auth 0 2024-03-29T10:50:33.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.461+0000 7fd0ff7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:50:33.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.461+0000 7fd0ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:33.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.461+0000 7fd0ff7fe640 1 -- 172.21.15.57:0/62720603 <== mon.2 v1:172.21.15.57:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd0f8001670 con 0x7fd1001457f0 2024-03-29T10:50:33.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.461+0000 7fd0ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:33.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.461+0000 7fd0ff7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:50:33.448 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.461+0000 7fd0ff7fe640 10 monclient: dump: 2024-03-29T10:50:33.448 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:50:33.448 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:33.448 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:33.449 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:33.449 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:33.449 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:33.449 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:50:33.449 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:50:33.449 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:50:33.449 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:33.449 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.463+0000 7fd1070ae640 5 monclient: authenticate success, global_id 4661 2024-03-29T10:50:33.449 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.463+0000 7fd1070ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:50:33.450 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.465+0000 7fd0ff7fe640 1 -- 172.21.15.57:0/62720603 <== mon.2 v1:172.21.15.57:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd0f80017d0 con 0x7fd1001457f0 2024-03-29T10:50:33.450 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.465+0000 7fd0ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:33.450 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.465+0000 7fd0ff7fe640 1 -- 172.21.15.57:0/62720603 <== mon.2 v1:172.21.15.57:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd0f8001c20 con 0x7fd1001457f0 2024-03-29T10:50:33.450 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.465+0000 7fd0ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:33.450 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.465+0000 7fd0ff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:50:33.450 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.465+0000 7fd0ff7fe640 10 monclient: dump: 2024-03-29T10:50:33.450 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:50:33.450 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:33.450 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:33.451 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:33.451 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:33.451 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:33.451 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:50:33.451 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:50:33.451 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:50:33.451 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:33.451 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.465+0000 7fd1070ae640 10 monclient: get_monmap_and_config success 2024-03-29T10:50:33.451 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.465+0000 7fd1070ae640 4 set_mon_vals no callback set 2024-03-29T10:50:33.451 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.466+0000 7fd1070ae640 10 monclient: shutdown 2024-03-29T10:50:33.451 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.466+0000 7fd1070ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:33.451 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.466+0000 7fd1070ae640 1 -- 172.21.15.57:0/62720603 >> v1:172.21.15.57:6790/0 conn(0x7fd1001457f0 legacy=0x7fd10013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:33.451 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.466+0000 7fd1070ae640 1 -- 172.21.15.57:0/62720603 shutdown_connections 2024-03-29T10:50:33.451 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.466+0000 7fd1070ae640 1 -- 172.21.15.57:0/62720603 >> 172.21.15.57:0/62720603 conn(0x7fd100066a00 msgr2=0x7fd100066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:33.452 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.467+0000 7fd1070ae640 1 -- 172.21.15.57:0/62720603 shutdown_connections 2024-03-29T10:50:33.452 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.468+0000 7fd1070ae640 1 -- 172.21.15.57:0/62720603 wait complete. 2024-03-29T10:50:33.452 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.468+0000 7fd1070ae640 10 monclient: build_initial_monmap 2024-03-29T10:50:33.452 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.469+0000 7fd1070ae640 10 monclient: monmap: 2024-03-29T10:50:33.452 INFO:teuthology.misc.health.smithi057.stderr:epoch 0 2024-03-29T10:50:33.453 INFO:teuthology.misc.health.smithi057.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:50:33.453 INFO:teuthology.misc.health.smithi057.stderr:last_changed 0.000000 2024-03-29T10:50:33.453 INFO:teuthology.misc.health.smithi057.stderr:created 0.000000 2024-03-29T10:50:33.453 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 0 (unknown) 2024-03-29T10:50:33.453 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:33.453 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.noname-a 2024-03-29T10:50:33.453 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.noname-b 2024-03-29T10:50:33.453 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.noname-c 2024-03-29T10:50:33.453 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:33.453 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.469+0000 7fd1070ae640 1 librados: starting msgr at 2024-03-29T10:50:33.453 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.469+0000 7fd1070ae640 1 librados: starting objecter 2024-03-29T10:50:33.453 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.469+0000 7fd1070ae640 1 Processor -- start 2024-03-29T10:50:33.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.469+0000 7fd1070ae640 1 -- start start 2024-03-29T10:50:33.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.469+0000 7fd1070ae640 1 librados: setting wanted keys 2024-03-29T10:50:33.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.469+0000 7fd1070ae640 1 librados: calling monclient init 2024-03-29T10:50:33.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.469+0000 7fd1070ae640 10 monclient: init 2024-03-29T10:50:33.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.470+0000 7fd1070ae640 10 monclient: _reopen_session rank -1 2024-03-29T10:50:33.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.470+0000 7fd1070ae640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:50:33.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.470+0000 7fd1070ae640 10 monclient(hunting): picked mon.noname-b con 0x7fd1000a9d70 addr v1:172.21.15.179:6789/0 2024-03-29T10:50:33.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.470+0000 7fd1070ae640 10 monclient(hunting): picked mon.noname-c con 0x7fd10013f650 addr v1:172.21.15.57:6790/0 2024-03-29T10:50:33.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.470+0000 7fd1070ae640 10 monclient(hunting): picked mon.noname-a con 0x7fd1001457f0 addr v1:172.21.15.57:6789/0 2024-03-29T10:50:33.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.470+0000 7fd1070ae640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1000acfb0 con 0x7fd1001457f0 2024-03-29T10:50:33.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.470+0000 7fd1070ae640 1 -- --> v1:172.21.15.179:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1001d70c0 con 0x7fd1000a9d70 2024-03-29T10:50:33.454 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.470+0000 7fd1070ae640 1 -- --> v1:172.21.15.57:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd10013fda0 con 0x7fd10013f650 2024-03-29T10:50:33.455 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.470+0000 7fd1070ae640 10 monclient(hunting): _renew_subs 2024-03-29T10:50:33.455 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.470+0000 7fd1070ae640 10 monclient(hunting): authenticate will time out at 1585.647806s 2024-03-29T10:50:33.456 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.473+0000 7fd0fffff640 1 --1- >> v1:172.21.15.57:6790/0 conn(0x7fd10013f650 0x7fd1001d75e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.57:6790/0 says I am v1:172.21.15.57:41523/0 (socket says 172.21.15.57:41523) 2024-03-29T10:50:33.456 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.473+0000 7fd0fffff640 1 -- 172.21.15.57:0/1894141888 learned_addr learned my addr 172.21.15.57:0/1894141888 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-29T10:50:33.456 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.473+0000 7fd0fd7fa640 10 client.?.objecter ms_handle_connect 0x7fd1001457f0 2024-03-29T10:50:33.456 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.473+0000 7fd0fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:33.456 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.473+0000 7fd0fd7fa640 10 client.?.objecter ms_handle_connect 0x7fd1000a9d70 2024-03-29T10:50:33.456 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.473+0000 7fd0fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:33.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.474+0000 7fd0fd7fa640 10 client.?.objecter ms_handle_connect 0x7fd10013f650 2024-03-29T10:50:33.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.474+0000 7fd0fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:50:33.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.474+0000 7fd0fd7fa640 1 -- 172.21.15.57:0/1894141888 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274951035 0 0) 0x7fd1000acfb0 con 0x7fd1001457f0 2024-03-29T10:50:33.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.474+0000 7fd0fd7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:50:33.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.474+0000 7fd0fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:33.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.474+0000 7fd0fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:33.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.474+0000 7fd0fd7fa640 10 monclient(hunting): my global_id is 4599 2024-03-29T10:50:33.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.474+0000 7fd0fd7fa640 1 -- 172.21.15.57:0/1894141888 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0ec004eb0 con 0x7fd1001457f0 2024-03-29T10:50:33.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.474+0000 7fd0fd7fa640 1 -- 172.21.15.57:0/1894141888 <== mon.1 v1:172.21.15.179:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450318961 0 0) 0x7fd1001d70c0 con 0x7fd1000a9d70 2024-03-29T10:50:33.463 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.474+0000 7fd0fd7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:50:33.464 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.474+0000 7fd0fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:33.464 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.474+0000 7fd0fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:33.464 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.474+0000 7fd0fd7fa640 10 monclient(hunting): my global_id is 4603 2024-03-29T10:50:33.464 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.474+0000 7fd0fd7fa640 1 -- 172.21.15.57:0/1894141888 --> v1:172.21.15.179:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1000acfb0 con 0x7fd1000a9d70 2024-03-29T10:50:33.464 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.475+0000 7fd0fd7fa640 1 -- 172.21.15.57:0/1894141888 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 754545404 0 0) 0x7fd0ec004eb0 con 0x7fd1001457f0 2024-03-29T10:50:33.464 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.475+0000 7fd0fd7fa640 1 -- 172.21.15.57:0/1894141888 --> v1:172.21.15.57:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1001d70c0 con 0x7fd1001457f0 2024-03-29T10:50:33.464 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.475+0000 7fd0fd7fa640 1 -- 172.21.15.57:0/1894141888 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd0f0001df0 con 0x7fd1001457f0 2024-03-29T10:50:33.464 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.475+0000 7fd0fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:33.464 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.475+0000 7fd0fd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:50:33.464 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.475+0000 7fd0fd7fa640 10 monclient(hunting): dump: 2024-03-29T10:50:33.464 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:50:33.464 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:33.465 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:33.465 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:33.465 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:33.465 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:33.465 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:50:33.465 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:50:33.465 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:50:33.465 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:33.465 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.475+0000 7fd0fd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:50:33.465 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.475+0000 7fd0fd7fa640 1 -- 172.21.15.57:0/1894141888 <== mon.2 v1:172.21.15.57:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3937589336 0 0) 0x7fd10013fda0 con 0x7fd10013f650 2024-03-29T10:50:33.465 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.476+0000 7fd0fd7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:50:33.465 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.476+0000 7fd0fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:50:33.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.476+0000 7fd0fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:50:33.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.476+0000 7fd0fd7fa640 10 monclient(hunting): my global_id is 4664 2024-03-29T10:50:33.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.476+0000 7fd0fd7fa640 1 -- 172.21.15.57:0/1894141888 --> v1:172.21.15.57:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0ec004eb0 con 0x7fd10013f650 2024-03-29T10:50:33.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.476+0000 7fd0fd7fa640 1 -- 172.21.15.57:0/1894141888 <== mon.1 v1:172.21.15.179:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2246156086 0 0) 0x7fd1000acfb0 con 0x7fd1000a9d70 2024-03-29T10:50:33.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.476+0000 7fd0fd7fa640 1 -- 172.21.15.57:0/1894141888 --> v1:172.21.15.179:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd10013fda0 con 0x7fd1000a9d70 2024-03-29T10:50:33.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.476+0000 7fd0fd7fa640 1 -- 172.21.15.57:0/1894141888 <== mon.1 v1:172.21.15.179:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd0f8001ae0 con 0x7fd1000a9d70 2024-03-29T10:50:33.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.476+0000 7fd0fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:33.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.476+0000 7fd0fd7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:50:33.466 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.476+0000 7fd0fd7fa640 10 monclient(hunting): dump: 2024-03-29T10:50:33.466 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:50:33.466 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:33.466 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:33.466 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:33.467 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:33.467 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:33.467 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:50:33.467 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:50:33.467 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:50:33.467 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:33.467 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.476+0000 7fd0fd7fa640 1 -- 172.21.15.57:0/1894141888 <== mon.0 v1:172.21.15.57:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2128464924 0 0) 0x7fd1001d70c0 con 0x7fd1001457f0 2024-03-29T10:50:33.467 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.476+0000 7fd0fd7fa640 1 -- 172.21.15.57:0/1894141888 >> v1:172.21.15.57:6790/0 conn(0x7fd10013f650 legacy=0x7fd1001d75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:33.467 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.476+0000 7fd0fd7fa640 1 -- 172.21.15.57:0/1894141888 >> v1:172.21.15.179:6789/0 conn(0x7fd1000a9d70 legacy=0x7fd100142560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:33.467 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.476+0000 7fd0fd7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:50:33.467 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.476+0000 7fd0fd7fa640 1 monclient: found mon.a 2024-03-29T10:50:33.467 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.476+0000 7fd0fd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:50:33.467 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.476+0000 7fd0fd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:50:33.467 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.476+0000 7fd0fd7fa640 1 -- 172.21.15.57:0/1894141888 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd100142d20 con 0x7fd1001457f0 2024-03-29T10:50:33.467 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.476+0000 7fd0fd7fa640 10 monclient: _finish_auth 0 2024-03-29T10:50:33.468 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.476+0000 7fd0fd7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:50:33.468 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.476+0000 7fd0fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:50:33.468 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.477+0000 7fd1070ae640 5 monclient: authenticate success, global_id 4599 2024-03-29T10:50:33.468 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.477+0000 7fd1070ae640 10 monclient: _renew_subs 2024-03-29T10:50:33.468 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.477+0000 7fd1070ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:50:33.468 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.477+0000 7fd1070ae640 1 -- 172.21.15.57:0/1894141888 --> v1:172.21.15.57:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1000abac0 con 0x7fd1001457f0 2024-03-29T10:50:33.468 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.477+0000 7fd1070ae640 10 client.4599.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:50:33.468 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.477+0000 7fd1070ae640 10 monclient: _renew_subs 2024-03-29T10:50:33.468 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.477+0000 7fd1070ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:50:33.468 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.477+0000 7fd1070ae640 1 -- 172.21.15.57:0/1894141888 --> v1:172.21.15.57:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1001d7ed0 con 0x7fd1001457f0 2024-03-29T10:50:33.468 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.477+0000 7fd1070ae640 1 librados: init done 2024-03-29T10:50:33.468 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.478+0000 7fd0fd7fa640 1 -- 172.21.15.57:0/1894141888 <== mon.0 v1:172.21.15.57:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd0f0001f50 con 0x7fd1001457f0 2024-03-29T10:50:33.468 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.478+0000 7fd0fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:50:33.468 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.478+0000 7fd0fd7fa640 1 -- 172.21.15.57:0/1894141888 <== mon.0 v1:172.21.15.57:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3814645127 0 0) 0x7fd0f0004fb0 con 0x7fd1001457f0 2024-03-29T10:50:33.469 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.478+0000 7fd0fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:50:33.469 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.478+0000 7fd0fd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:50:33.469 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.478+0000 7fd0fd7fa640 10 monclient: dump: 2024-03-29T10:50:33.469 INFO:teuthology.misc.health.smithi057.stderr:epoch 1 2024-03-29T10:50:33.469 INFO:teuthology.misc.health.smithi057.stderr:fsid c698cedc-432d-480a-94d2-408defa62225 2024-03-29T10:50:33.469 INFO:teuthology.misc.health.smithi057.stderr:last_changed 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:33.469 INFO:teuthology.misc.health.smithi057.stderr:created 2024-03-29T10:38:53.590916+0000 2024-03-29T10:50:33.469 INFO:teuthology.misc.health.smithi057.stderr:min_mon_release 19 (squid) 2024-03-29T10:50:33.469 INFO:teuthology.misc.health.smithi057.stderr:election_strategy: 1 2024-03-29T10:50:33.469 INFO:teuthology.misc.health.smithi057.stderr:0: v1:172.21.15.57:6789/0 mon.a 2024-03-29T10:50:33.469 INFO:teuthology.misc.health.smithi057.stderr:1: v1:172.21.15.179:6789/0 mon.b 2024-03-29T10:50:33.469 INFO:teuthology.misc.health.smithi057.stderr:2: v1:172.21.15.57:6790/0 mon.c 2024-03-29T10:50:33.469 INFO:teuthology.misc.health.smithi057.stderr: 2024-03-29T10:50:33.469 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.479+0000 7fd0feffd640 4 set_mon_vals no callback set 2024-03-29T10:50:33.470 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.480+0000 7fd1070ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:33.470 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.480+0000 7fd1070ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:33.470 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.480+0000 7fd1070ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:50:33.470 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.480+0000 7fd1070ae640 1 -- 172.21.15.57:0/1894141888 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e0005080 con 0x7fd1001457f0 2024-03-29T10:50:33.470 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.480+0000 7fd0fd7fa640 1 -- 172.21.15.57:0/1894141888 <== mon.0 v1:172.21.15.57:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3948986387 0 0) 0x7fd0f001af80 con 0x7fd1001457f0 2024-03-29T10:50:33.471 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.481+0000 7fd0fd7fa640 1 -- 172.21.15.57:0/1894141888 <== mon.0 v1:172.21.15.57:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3486612304 0 0) 0x7fd0f0084d70 con 0x7fd1001457f0 2024-03-29T10:50:33.471 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.481+0000 7fd0fd7fa640 10 client.4599.objecter ms_dispatch 0x7fd100000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:50:33.471 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.481+0000 7fd0fd7fa640 3 client.4599.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:50:33.471 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.481+0000 7fd0fd7fa640 3 client.4599.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:50:33.471 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.481+0000 7fd0fd7fa640 20 client.4599.objecter dump_active .. 0 homeless 2024-03-29T10:50:33.471 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.483+0000 7fd104e4a640 10 monclient: get_auth_request con 0x7fd0ec06ded0 auth_method 0 2024-03-29T10:50:33.471 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.483+0000 7fd0fd7fa640 10 client.4599.objecter ms_handle_connect 0x7fd0ec06ded0 2024-03-29T10:50:33.472 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.488+0000 7fd0fd7fa640 1 -- 172.21.15.57:0/1894141888 <== mon.0 v1:172.21.15.57:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd0f0050650 con 0x7fd1001457f0 2024-03-29T10:50:33.472 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.488+0000 7fd0fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:50:33.472 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.488+0000 7fd0fd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:50:33.806 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.823+0000 7fd1070ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:33.806 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.823+0000 7fd1070ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:33.806 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.823+0000 7fd1070ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.57:6789/0 2024-03-29T10:50:33.806 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.823+0000 7fd1070ae640 1 -- 172.21.15.57:0/1894141888 --> v1:172.21.15.57:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0e0005330 con 0x7fd1001457f0 2024-03-29T10:50:33.808 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.826+0000 7fd0fd7fa640 1 -- 172.21.15.57:0/1894141888 <== mon.0 v1:172.21.15.57:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2357246973) 0x7fd0f0052580 con 0x7fd1001457f0 2024-03-29T10:50:33.808 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.826+0000 7fd0fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:50:33.808 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.826+0000 7fd0fd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:50:33.808 INFO:teuthology.misc.health.smithi057.stdout: 2024-03-29T10:50:33.809 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":11,"fsid":"c698cedc-432d-480a-94d2-408defa62225","created":"2024-03-29T10:39:35.164324+0000","modified":"2024-03-29T10:49:40.047000+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:49:40.047000+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c4d60fc-22f9-4e71-93e8-b5cb43c0b5c4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e1569630-d46c-4cc7-93c7-73c7ad2140ac","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"04c8ad6d-9d01-4655-80f4-334cb8e4f90e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"5566b3b8-270a-45b8-a6ba-f5f0228937e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a853ea6e-33dd-42f3-a7a9-0d424e82f27d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2d808cb3-4df1-4247-a61b-7c32fe4954ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2c2faf76-845d-4a6d-a834-faac22103ef3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e42d7620-c54f-43e2-96e7-86680f5b7c70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:50:33.812 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.829+0000 7fd1070ae640 10 librados: watch_flush enter 2024-03-29T10:50:33.812 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.829+0000 7fd1070ae640 10 librados: watch_flush exit 2024-03-29T10:50:33.812 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.830+0000 7fd1070ae640 20 client.4599.objecter shutdown clearing up homeless session... 2024-03-29T10:50:33.813 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.830+0000 7fd1070ae640 10 client.4599.objecter successfully canceled tick 2024-03-29T10:50:33.813 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.830+0000 7fd1070ae640 1 -- 172.21.15.57:0/1894141888 >> v1:172.21.15.179:6816/2698295936 conn(0x7fd0ec06ded0 legacy=0x7fd0ec070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:33.813 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.830+0000 7fd1070ae640 10 monclient: shutdown 2024-03-29T10:50:33.813 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.830+0000 7fd1070ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:50:33.813 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.830+0000 7fd1070ae640 1 -- 172.21.15.57:0/1894141888 >> v1:172.21.15.57:6789/0 conn(0x7fd1001457f0 legacy=0x7fd100137d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:50:33.813 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.830+0000 7fd1070ae640 1 -- 172.21.15.57:0/1894141888 shutdown_connections 2024-03-29T10:50:33.813 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.830+0000 7fd1070ae640 1 -- 172.21.15.57:0/1894141888 >> 172.21.15.57:0/1894141888 conn(0x7fd100066a00 msgr2=0x7fd1000b4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:50:33.816 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.833+0000 7fd1070ae640 1 -- 172.21.15.57:0/1894141888 shutdown_connections 2024-03-29T10:50:33.816 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.833+0000 7fd1070ae640 1 -- 172.21.15.57:0/1894141888 wait complete. 2024-03-29T10:50:33.816 INFO:teuthology.misc.health.smithi057.stderr:2024-03-29T10:50:33.833+0000 7fd1070ae640 1 librados: shutdown 2024-03-29T10:50:33.833 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:50:33.833 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:50:33.835 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T10:50:33.835 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T10:50:33.835 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:33.870 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T10:50:33.870 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T10:50:33.870 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:33.948 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T10:50:33.948 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T10:50:33.948 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:34.017 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T10:50:34.017 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T10:50:34.017 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:34.079 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T10:50:34.080 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T10:50:34.080 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:34.122 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T10:50:34.123 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T10:50:34.123 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:34.155 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T10:50:34.155 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T10:50:34.155 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:34.224 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T10:50:34.225 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T10:50:34.225 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:34.291 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T10:50:34.291 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T10:50:34.292 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T10:50:34.292 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:34.316 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T10:50:34.316 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T10:50:34.316 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:34.357 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T10:50:34.357 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T10:50:34.357 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T10:50:34.358 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:34.369 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T10:50:34.369 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T10:50:34.369 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:34.420 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T10:50:34.420 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T10:50:34.421 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:50:34.431 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T10:50:34.432 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T10:50:34.432 DEBUG:teuthology.orchestra.run.smithi057:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T10:50:34.490 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi057.front.sepia.ceph.com 2024-03-29T10:50:34.491 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T10:50:36.632 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi057.front.sepia.ceph.com 2024-03-29T10:50:36.633 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T10:50:36.736 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi057.front.sepia.ceph.com 2024-03-29T10:50:36.736 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T10:50:36.804 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi057.front.sepia.ceph.com 2024-03-29T10:50:36.804 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T10:50:36.986 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi179.front.sepia.ceph.com 2024-03-29T10:50:36.986 DEBUG:teuthology.orchestra.run.smithi179:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T10:50:40.505 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi179.front.sepia.ceph.com 2024-03-29T10:50:40.505 DEBUG:teuthology.orchestra.run.smithi179:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T10:50:40.590 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi179.front.sepia.ceph.com 2024-03-29T10:50:40.590 DEBUG:teuthology.orchestra.run.smithi179:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T10:50:40.656 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi179.front.sepia.ceph.com 2024-03-29T10:50:40.656 DEBUG:teuthology.orchestra.run.smithi179:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T10:50:40.707 INFO:tasks.ceph:Archiving mon data... 2024-03-29T10:50:40.709 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622641/data/mon.a.tgz 2024-03-29T10:50:40.709 DEBUG:teuthology.orchestra.run.smithi057:> mktemp 2024-03-29T10:50:40.728 INFO:teuthology.orchestra.run.smithi057.stdout:/tmp/tmp.PVkW9l7cK9 2024-03-29T10:50:40.728 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.PVkW9l7cK9 2024-03-29T10:50:40.852 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0666 /tmp/tmp.PVkW9l7cK9 2024-03-29T10:50:40.941 DEBUG:teuthology.orchestra.remote:smithi057:/tmp/tmp.PVkW9l7cK9 is 213KB 2024-03-29T10:50:41.036 DEBUG:teuthology.orchestra.run.smithi057:> rm -fr /tmp/tmp.PVkW9l7cK9 2024-03-29T10:50:41.052 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622641/data/mon.c.tgz 2024-03-29T10:50:41.052 DEBUG:teuthology.orchestra.run.smithi057:> mktemp 2024-03-29T10:50:41.106 INFO:teuthology.orchestra.run.smithi057.stdout:/tmp/tmp.pSGK5JinS8 2024-03-29T10:50:41.107 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.pSGK5JinS8 2024-03-29T10:50:41.211 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0666 /tmp/tmp.pSGK5JinS8 2024-03-29T10:50:41.296 DEBUG:teuthology.orchestra.remote:smithi057:/tmp/tmp.pSGK5JinS8 is 213KB 2024-03-29T10:50:41.390 DEBUG:teuthology.orchestra.run.smithi057:> rm -fr /tmp/tmp.pSGK5JinS8 2024-03-29T10:50:41.404 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622641/data/mon.b.tgz 2024-03-29T10:50:41.405 DEBUG:teuthology.orchestra.run.smithi179:> mktemp 2024-03-29T10:50:41.423 INFO:teuthology.orchestra.run.smithi179.stdout:/tmp/tmp.CWYvq5QQGY 2024-03-29T10:50:41.423 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.CWYvq5QQGY 2024-03-29T10:50:41.544 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0666 /tmp/tmp.CWYvq5QQGY 2024-03-29T10:50:41.633 DEBUG:teuthology.orchestra.remote:smithi179:/tmp/tmp.CWYvq5QQGY is 213KB 2024-03-29T10:50:41.728 DEBUG:teuthology.orchestra.run.smithi179:> rm -fr /tmp/tmp.CWYvq5QQGY 2024-03-29T10:50:41.743 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T10:50:41.743 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T10:50:41.746 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T10:50:41.827 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T10:50:41.865 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T10:50:41.865 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T10:50:41.867 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622641/remote/smithi057/crash 2024-03-29T10:50:41.868 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T10:50:41.895 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622641/remote/smithi179/crash 2024-03-29T10:50:41.896 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T10:50:41.932 INFO:tasks.ceph:Compressing logs... 2024-03-29T10:50:41.932 DEBUG:teuthology.orchestra.run.smithi057:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T10:50:41.939 DEBUG:teuthology.orchestra.run.smithi179:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T10:50:41.964 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T10:50:42.003 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41494.log 2024-03-29T10:50:42.003 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39345.log 2024-03-29T10:50:42.003 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.41494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38327.log 2024-03-29T10:50:42.003 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.39345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40831.log 2024-03-29T10:50:42.003 INFO:teuthology.orchestra.run.smithi057.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41494.log.gz 2024-03-29T10:50:42.003 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.38327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39267.log 2024-03-29T10:50:42.003 INFO:teuthology.orchestra.run.smithi057.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39345.log.gz 2024-03-29T10:50:42.004 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39462.log 2024-03-29T10:50:42.004 INFO:teuthology.orchestra.run.smithi057.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38327.log.gz 2024-03-29T10:50:42.004 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.39267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38877.log 2024-03-29T10:50:42.004 INFO:teuthology.orchestra.run.smithi057.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40831.log.gz 2024-03-29T10:50:42.004 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.39462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41182.log 2024-03-29T10:50:42.004 INFO:teuthology.orchestra.run.smithi057.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39267.log.gz 2024-03-29T10:50:42.004 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.38877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40987.log 2024-03-29T10:50:42.004 INFO:teuthology.orchestra.run.smithi057.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.39462.log.gz 2024-03-29T10:50:42.004 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.41182.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38877.log.gz 2024-03-29T10:50:42.004 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40168.log 2024-03-29T10:50:42.004 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40246.log 2024-03-29T10:50:42.004 INFO:teuthology.orchestra.run.smithi057.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41182.log.gz 2024-03-29T10:50:42.004 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40168.log: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.40987.log.gz 2024-03-29T10:50:42.004 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41104.log 2024-03-29T10:50:42.005 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40870.log 2024-03-29T10:50:42.005 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40246.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40168.log.gz 2024-03-29T10:50:42.005 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.41104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38524.log 2024-03-29T10:50:42.005 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40246.log.gz 2024-03-29T10:50:42.005 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39306.log 2024-03-29T10:50:42.005 INFO:teuthology.orchestra.run.smithi057.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41104.log.gz 2024-03-29T10:50:42.005 INFO:teuthology.orchestra.run.smithi057.stderr:gzip/var/log/ceph/ceph-client.admin.38524.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39895.log 2024-03-29T10:50:42.005 INFO:teuthology.orchestra.run.smithi057.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40870.log.gz 2024-03-29T10:50:42.005 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.39306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39072.log 2024-03-29T10:50:42.005 INFO:teuthology.orchestra.run.smithi057.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38524.log.gz 2024-03-29T10:50:42.005 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.39895.log: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.39306.log.gz 2024-03-29T10:50:42.005 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41650.log 2024-03-29T10:50:42.005 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.39072.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39895.log.gz 2024-03-29T10:50:42.005 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39934.log 2024-03-29T10:50:42.005 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.41650.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39072.log.gz 2024-03-29T10:50:42.006 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40909.log 2024-03-29T10:50:42.006 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.39934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38799.log 2024-03-29T10:50:42.006 INFO:teuthology.orchestra.run.smithi057.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41650.log.gz 2024-03-29T10:50:42.006 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41455.log 2024-03-29T10:50:42.006 INFO:teuthology.orchestra.run.smithi057.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39934.log.gz 2024-03-29T10:50:42.006 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.38799.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40909.log.gz 2024-03-29T10:50:42.006 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39501.log 2024-03-29T10:50:42.006 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.41455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38994.log 2024-03-29T10:50:42.006 INFO:teuthology.orchestra.run.smithi057.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38799.log.gz 2024-03-29T10:50:42.006 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.39501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40012.log 2024-03-29T10:50:42.006 INFO:teuthology.orchestra.run.smithi057.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41455.log.gz 2024-03-29T10:50:42.006 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.38994.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39501.log.gz 2024-03-29T10:50:42.006 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34582.log 2024-03-29T10:50:42.006 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40012.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T10:50:42.006 INFO:teuthology.orchestra.run.smithi057.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38994.log.gz 2024-03-29T10:50:42.007 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.34582.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28751.log 86.5% 2024-03-29T10:50:42.007 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40012.log.gz 2024-03-29T10:50:42.007 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39540.log 2024-03-29T10:50:42.007 INFO:teuthology.orchestra.run.smithi057.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.34582.log.gz 2024-03-29T10:50:42.007 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph.tmp-client.admin.28751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39735.log 2024-03-29T10:50:42.007 INFO:teuthology.orchestra.run.smithi057.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28751.log.gz 2024-03-29T10:50:42.007 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.39540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40051.log 2024-03-29T10:50:42.007 INFO:teuthology.orchestra.run.smithi057.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T10:50:42.007 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.39735.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39540.log.gz 2024-03-29T10:50:42.007 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38604.log 2024-03-29T10:50:42.007 INFO:teuthology.orchestra.run.smithi057.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39735.log.gz 2024-03-29T10:50:42.007 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40363.log 2024-03-29T10:50:42.007 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.38604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39813.log 2024-03-29T10:50:42.007 INFO:teuthology.orchestra.run.smithi057.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40051.log.gz 2024-03-29T10:50:42.007 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40363.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38604.log.gz 2024-03-29T10:50:42.008 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41143.log 2024-03-29T10:50:42.008 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.39813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40558.log 2024-03-29T10:50:42.008 INFO:teuthology.orchestra.run.smithi057.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40363.log.gz 2024-03-29T10:50:42.008 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.41143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41065.log 2024-03-29T10:50:42.008 INFO:teuthology.orchestra.run.smithi057.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39813.log.gz 2024-03-29T10:50:42.008 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40558.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41143.log.gz 2024-03-29T10:50:42.008 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39657.log 2024-03-29T10:50:42.008 INFO:teuthology.orchestra.run.smithi057.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40558.log.gz 2024-03-29T10:50:42.008 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.41065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40714.log 2024-03-29T10:50:42.008 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.39657.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T10:50:42.008 INFO:teuthology.orchestra.run.smithi057.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.41065.log.gz 2024-03-29T10:50:42.008 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40402.log 2024-03-29T10:50:42.008 INFO:teuthology.orchestra.run.smithi057.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39657.log.gz 2024-03-29T10:50:42.008 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-osd.1.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40714.log.gz 2024-03-29T10:50:42.008 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39384.log 2024-03-29T10:50:42.009 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39228.log 2024-03-29T10:50:42.009 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.39384.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40402.log.gz 2024-03-29T10:50:42.009 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40480.log 2024-03-29T10:50:42.009 INFO:teuthology.orchestra.run.smithi057.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.39384.log.gz 2024-03-29T10:50:42.009 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.39228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41221.log 2024-03-29T10:50:42.009 INFO:teuthology.orchestra.run.smithi057.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39228.log.gz 2024-03-29T10:50:42.009 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40675.log 2024-03-29T10:50:42.009 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40480.log: /var/log/ceph/ceph-client.admin.41221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40324.log 2024-03-29T10:50:42.009 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40675.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40480.log.gz 2024-03-29T10:50:42.009 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34976.log 2024-03-29T10:50:42.009 INFO:teuthology.orchestra.run.smithi057.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41221.log.gz 2024-03-29T10:50:42.009 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40324.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40675.log.gz 2024-03-29T10:50:42.009 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38249.log 2024-03-29T10:50:42.009 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.34976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38366.log 2024-03-29T10:50:42.009 INFO:teuthology.orchestra.run.smithi057.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40324.log.gz 2024-03-29T10:50:42.010 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.38249.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34976.log.gz 2024-03-29T10:50:42.010 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40636.log 2024-03-29T10:50:42.010 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.38366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41299.log 2024-03-29T10:50:42.010 INFO:teuthology.orchestra.run.smithi057.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38249.log.gz 2024-03-29T10:50:42.010 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39973.log 2024-03-29T10:50:42.010 INFO:teuthology.orchestra.run.smithi057.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38366.log.gz 2024-03-29T10:50:42.010 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.41299.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40636.log.gz 2024-03-29T10:50:42.010 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38288.log 2024-03-29T10:50:42.010 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.39973.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41299.log.gz 2024-03-29T10:50:42.010 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40441.log 2024-03-29T10:50:42.010 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.38288.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39973.log.gz 2024-03-29T10:50:42.010 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38682.log 2024-03-29T10:50:42.010 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40441.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38288.log.gz 2024-03-29T10:50:42.010 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T10:50:42.010 INFO:teuthology.orchestra.run.smithi057.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40441.log.gz 2024-03-29T10:50:42.011 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.38682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40753.log 2024-03-29T10:50:42.011 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38485.log 2024-03-29T10:50:42.011 INFO:teuthology.orchestra.run.smithi057.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38682.log.gz 2024-03-29T10:50:42.011 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39423.log 2024-03-29T10:50:42.011 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.38485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41260.log 2024-03-29T10:50:42.011 INFO:teuthology.orchestra.run.smithi057.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40753.log.gz 2024-03-29T10:50:42.011 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.39423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39852.log 2024-03-29T10:50:42.011 INFO:teuthology.orchestra.run.smithi057.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38485.log.gz 2024-03-29T10:50:42.011 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.41260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39189.log 2024-03-29T10:50:42.011 INFO:teuthology.orchestra.run.smithi057.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39423.log.gz 2024-03-29T10:50:42.011 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.39852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40792.log 2024-03-29T10:50:42.011 INFO:teuthology.orchestra.run.smithi057.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41260.log.gz 2024-03-29T10:50:42.011 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.39189.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39852.log.gz 2024-03-29T10:50:42.011 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39150.log 2024-03-29T10:50:42.012 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39111.log 2024-03-29T10:50:42.012 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40792.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39189.log.gz 2024-03-29T10:50:42.012 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.39150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38643.log 2024-03-29T10:50:42.012 INFO:teuthology.orchestra.run.smithi057.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40792.log.gz 2024-03-29T10:50:42.012 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.39111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39774.log 2024-03-29T10:50:42.012 INFO:teuthology.orchestra.run.smithi057.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39150.log.gz 2024-03-29T10:50:42.012 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.38643.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T10:50:42.012 INFO:teuthology.orchestra.run.smithi057.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.39111.log.gz 2024-03-29T10:50:42.012 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.39774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41611.log 2024-03-29T10:50:42.012 INFO:teuthology.orchestra.run.smithi057.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38643.log.gz 2024-03-29T10:50:42.012 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40285.log 2024-03-29T10:50:42.012 INFO:teuthology.orchestra.run.smithi057.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39774.log.gz 2024-03-29T10:50:42.012 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.41611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34877.log 2024-03-29T10:50:42.012 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40285.log: gzip -5 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41611.log.gz 2024-03-29T10:50:42.012 INFO:teuthology.orchestra.run.smithi057.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40597.log 2024-03-29T10:50:42.012 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.34877.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40285.log.gz 2024-03-29T10:50:42.013 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38760.log 2024-03-29T10:50:42.013 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40597.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34877.log.gz 2024-03-29T10:50:42.013 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40090.log 2024-03-29T10:50:42.013 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.38760.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40597.log.gz 2024-03-29T10:50:42.013 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41572.log 2024-03-29T10:50:42.013 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40090.log: gzip 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38760.log.gz 2024-03-29T10:50:42.013 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41026.log 2024-03-29T10:50:42.013 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.41572.log: 85.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38210.log 2024-03-29T10:50:42.013 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40090.log.gz 2024-03-29T10:50:42.013 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.41026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38563.log 2024-03-29T10:50:42.013 INFO:teuthology.orchestra.run.smithi057.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41572.log.gz 2024-03-29T10:50:42.013 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.38210.log: -- /var/log/ceph/ceph-osd.0.log 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41026.log.gz 2024-03-29T10:50:42.013 INFO:teuthology.orchestra.run.smithi057.stderr: 2024-03-29T10:50:42.013 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.38563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39579.log 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38210.log.gz 2024-03-29T10:50:42.013 INFO:teuthology.orchestra.run.smithi057.stderr: 2024-03-29T10:50:42.013 INFO:teuthology.orchestra.run.smithi057.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38563.log.gz 2024-03-29T10:50:42.014 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40948.log 2024-03-29T10:50:42.014 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.39579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41533.log 2024-03-29T10:50:42.014 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40948.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39579.log.gz 2024-03-29T10:50:42.014 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41416.log 2024-03-29T10:50:42.014 INFO:teuthology.orchestra.run.smithi057.stderr:gzip 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40948.log.gz 2024-03-29T10:50:42.014 INFO:teuthology.orchestra.run.smithi057.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40207.log 2024-03-29T10:50:42.014 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.41416.log: /var/log/ceph/ceph-client.admin.41533.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T10:50:42.014 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40207.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41416.log.gz 2024-03-29T10:50:42.014 INFO:teuthology.orchestra.run.smithi057.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41533.log.gz 2024-03-29T10:50:42.014 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41377.log 2024-03-29T10:50:42.014 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40207.log.gz 2024-03-29T10:50:42.014 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39696.log 2024-03-29T10:50:42.014 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph.log: gzip/var/log/ceph/ceph-client.admin.41377.log: -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T10:50:42.014 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.39696.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41377.log.gz 2024-03-29T10:50:42.015 INFO:teuthology.orchestra.run.smithi057.stderr: 86.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.39696.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.38838.log 2024-03-29T10:50:42.015 INFO:teuthology.orchestra.run.smithi057.stderr: 2024-03-29T10:50:42.015 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-osd.2.log: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T10:50:42.015 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38405.log 2024-03-29T10:50:42.015 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38916.log 2024-03-29T10:50:42.015 INFO:teuthology.orchestra.run.smithi057.stderr: 93.7%/var/log/ceph/ceph-client.admin.38838.log: -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T10:50:42.015 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.38405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41338.log 2024-03-29T10:50:42.015 INFO:teuthology.orchestra.run.smithi057.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38838.log.gz 2024-03-29T10:50:42.015 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.38916.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38405.log.gz 2024-03-29T10:50:42.015 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40129.log 2024-03-29T10:50:42.015 INFO:teuthology.orchestra.run.smithi057.stderr: 86.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38444.log 2024-03-29T10:50:42.015 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38916.log.gz 2024-03-29T10:50:42.015 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.41338.log: /var/log/ceph/ceph-client.admin.40129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40519.log 2024-03-29T10:50:42.015 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.38444.log: 85.4% 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41338.log.gz 2024-03-29T10:50:42.016 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40129.log.gz 2024-03-29T10:50:42.016 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39033.log 2024-03-29T10:50:42.016 INFO:teuthology.orchestra.run.smithi057.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38444.log.gz 2024-03-29T10:50:42.016 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39618.log 2024-03-29T10:50:42.016 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.40519.log: /var/log/ceph/ceph-client.admin.39033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38721.log 2024-03-29T10:50:42.016 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.39618.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40519.log.gz 2024-03-29T10:50:42.016 INFO:teuthology.orchestra.run.smithi057.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39033.log.gz 2024-03-29T10:50:42.016 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39618.log.gz 2024-03-29T10:50:42.017 INFO:teuthology.orchestra.run.smithi057.stderr: -- /var/log/ceph/ceph-client.admin.38955.log 2024-03-29T10:50:42.017 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.38721.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T10:50:42.017 INFO:teuthology.orchestra.run.smithi057.stderr: 86.4%/var/log/ceph/ceph-client.admin.38955.log: -- replaced with /var/log/ceph/ceph-client.admin.38721.log.gz 2024-03-29T10:50:42.017 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-mon.c.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38955.log.gz 2024-03-29T10:50:42.022 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T10:50:42.023 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34847.log 2024-03-29T10:50:42.023 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T10:50:42.023 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34847.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T10:50:42.023 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34807.log 2024-03-29T10:50:42.023 INFO:teuthology.orchestra.run.smithi179.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34847.log.gz 2024-03-29T10:50:42.023 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34887.log 2024-03-29T10:50:42.023 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-osd.4.log: 88.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T10:50:42.023 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.34807.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T10:50:42.023 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T10:50:42.023 INFO:teuthology.orchestra.run.smithi179.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34807.log.gz/var/log/ceph/ceph-client.admin.34887.log: 2024-03-29T10:50:42.023 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34927.log 2024-03-29T10:50:42.023 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-osd.5.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34887.log.gz 2024-03-29T10:50:42.023 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T10:50:42.023 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.34927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34727.log 2024-03-29T10:50:42.024 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34767.log 2024-03-29T10:50:42.024 INFO:teuthology.orchestra.run.smithi179.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34927.log.gz 2024-03-29T10:50:42.024 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.34727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35007.log 2024-03-29T10:50:42.024 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-client.admin.34767.log: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T10:50:42.024 INFO:teuthology.orchestra.run.smithi179.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34727.log.gz 2024-03-29T10:50:42.024 INFO:teuthology.orchestra.run.smithi179.stderr: 87.3%/var/log/ceph/ceph-client.admin.35007.log: -- replaced with /var/log/ceph/ceph-client.admin.34767.log.gz 2024-03-29T10:50:42.024 INFO:teuthology.orchestra.run.smithi179.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35007.log.gz 2024-03-29T10:50:42.024 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T10:50:42.024 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34967.log 2024-03-29T10:50:42.024 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.34967.log: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T10:50:42.024 INFO:teuthology.orchestra.run.smithi179.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34967.log.gz 2024-03-29T10:50:42.078 INFO:teuthology.orchestra.run.smithi179.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T10:50:53.561 INFO:teuthology.orchestra.run.smithi179.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T10:50:53.660 INFO:teuthology.orchestra.run.smithi179.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T10:50:53.728 INFO:teuthology.orchestra.run.smithi179.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T10:50:54.032 INFO:teuthology.orchestra.run.smithi057.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T10:50:57.825 INFO:teuthology.orchestra.run.smithi179.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T10:50:58.068 INFO:teuthology.orchestra.run.smithi057.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T10:50:58.115 INFO:teuthology.orchestra.run.smithi057.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T10:50:59.770 INFO:teuthology.orchestra.run.smithi057.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T10:51:31.107 INFO:teuthology.orchestra.run.smithi057.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T10:51:33.495 INFO:teuthology.orchestra.run.smithi057.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T10:51:33.497 INFO:teuthology.orchestra.run.smithi057.stderr: 2024-03-29T10:51:33.497 INFO:teuthology.orchestra.run.smithi057.stderr:real 0m51.545s 2024-03-29T10:51:33.497 INFO:teuthology.orchestra.run.smithi057.stderr:user 2m37.431s 2024-03-29T10:51:33.497 INFO:teuthology.orchestra.run.smithi057.stderr:sys 0m5.047s 2024-03-29T10:51:53.268 INFO:teuthology.orchestra.run.smithi179.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T10:51:53.269 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-29T10:51:53.270 INFO:teuthology.orchestra.run.smithi179.stderr:real 1m11.279s 2024-03-29T10:51:53.270 INFO:teuthology.orchestra.run.smithi179.stderr:user 1m57.455s 2024-03-29T10:51:53.270 INFO:teuthology.orchestra.run.smithi179.stderr:sys 0m4.442s 2024-03-29T10:51:53.270 INFO:tasks.ceph:Archiving logs... 2024-03-29T10:51:53.270 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622641/remote/smithi057/log 2024-03-29T10:51:53.271 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T10:52:16.099 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622641/remote/smithi179/log 2024-03-29T10:52:16.101 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T10:52:34.770 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:52:34.991 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f9aaef0fd383456a9511deef9a1dbe1b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:52:34.995 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T10:52:35.004 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T10:52:35.185 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T10:52:35.185 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T10:52:35.200 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T10:52:35.284 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T10:52:35.328 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T10:52:35.328 DEBUG:teuthology.orchestra.run.smithi057:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:52:35.331 DEBUG:teuthology.orchestra.run.smithi179:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:52:35.345 INFO:teuthology.orchestra.run.smithi057.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:52:35.348 INFO:teuthology.orchestra.run.smithi179.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:52:35.349 INFO:teuthology.orchestra.run.smithi057.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:52:35.350 INFO:teuthology.orchestra.run.smithi057.stdout:=============================================================================== 2024-03-29T10:52:35.350 INFO:teuthology.orchestra.run.smithi057.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 19 +425us[ +425us] +/- 71ms 2024-03-29T10:52:35.350 INFO:teuthology.orchestra.run.smithi057.stdout:^* hv02.front.sepia.ceph.com 3 6 377 19 -185us[ -186us] +/- 51ms 2024-03-29T10:52:35.350 INFO:teuthology.orchestra.run.smithi057.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 -201us[ -202us] +/- 82ms 2024-03-29T10:52:35.350 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:52:35.352 INFO:teuthology.orchestra.run.smithi179.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:52:35.352 INFO:teuthology.orchestra.run.smithi179.stdout:=============================================================================== 2024-03-29T10:52:35.353 INFO:teuthology.orchestra.run.smithi179.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 21 +425us[ +424us] +/- 71ms 2024-03-29T10:52:35.353 INFO:teuthology.orchestra.run.smithi179.stdout:^* hv02.front.sepia.ceph.com 3 6 377 19 -185us[ -185us] +/- 51ms 2024-03-29T10:52:35.353 INFO:teuthology.orchestra.run.smithi179.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 -193us[ -194us] +/- 82ms 2024-03-29T10:52:35.353 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:52:35.354 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T10:52:35.396 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T10:52:35.397 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T10:52:35.480 DEBUG:teuthology.orchestra.run.smithi057:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T10:52:35.553 DEBUG:teuthology.orchestra.run.smithi179:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T10:52:35.624 DEBUG:teuthology.orchestra.run.smithi057:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:52:35.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:52:35.658 DEBUG:teuthology.orchestra.run.smithi179:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:52:35.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:52:35.690 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T10:52:35.699 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T10:52:35.766 INFO:teuthology.task.internal:Duration was 1389.215081 seconds 2024-03-29T10:52:35.766 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T10:52:35.893 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T10:52:35.894 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T10:52:35.897 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T10:52:35.944 INFO:teuthology.orchestra.run.smithi057.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:52:35.944 INFO:teuthology.orchestra.run.smithi179.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:52:36.339 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T10:52:36.339 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi057.front.sepia.ceph.com 2024-03-29T10:52:36.340 DEBUG:teuthology.orchestra.run.smithi057:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T10:52:36.362 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi179.front.sepia.ceph.com 2024-03-29T10:52:36.363 DEBUG:teuthology.orchestra.run.smithi179:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T10:52:36.389 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T10:52:36.389 DEBUG:teuthology.orchestra.run.smithi057:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T10:52:36.404 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T10:52:36.462 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T10:52:36.462 DEBUG:teuthology.orchestra.run.smithi057:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T10:52:36.477 DEBUG:teuthology.orchestra.run.smithi179:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T10:52:36.599 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T10:52:36.609 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T10:52:36.609 DEBUG:teuthology.orchestra.run.smithi057:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T10:52:36.637 DEBUG:teuthology.orchestra.run.smithi179:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T10:52:36.666 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T10:52:36.678 DEBUG:teuthology.orchestra.run.smithi057:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:52:36.680 DEBUG:teuthology.orchestra.run.smithi179:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:52:36.704 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern = core 2024-03-29T10:52:36.732 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = core 2024-03-29T10:52:36.768 DEBUG:teuthology.orchestra.run.smithi057:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:52:36.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:52:36.791 DEBUG:teuthology.orchestra.run.smithi179:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:52:36.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:52:36.822 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T10:52:36.833 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T10:52:36.834 DEBUG:teuthology.misc:Transferring archived files from smithi057:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622641/remote/smithi057 2024-03-29T10:52:36.834 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T10:52:36.870 DEBUG:teuthology.misc:Transferring archived files from smithi179:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622641/remote/smithi179 2024-03-29T10:52:36.870 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T10:52:36.908 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T10:52:36.908 DEBUG:teuthology.orchestra.run.smithi057:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T10:52:36.910 DEBUG:teuthology.orchestra.run.smithi179:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T10:52:36.958 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T10:52:36.969 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T10:52:36.969 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T10:52:37.125 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T10:52:37.126 DEBUG:teuthology.orchestra.run.smithi057:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T10:52:37.128 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T10:52:37.140 INFO:teuthology.orchestra.run.smithi057.stdout: 262170 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 10:52 /home/ubuntu/cephtest 2024-03-29T10:52:37.143 INFO:teuthology.orchestra.run.smithi179.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 10:52 /home/ubuntu/cephtest 2024-03-29T10:52:37.144 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T10:52:37.258 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T10:52:37.531 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T10:52:37.560 DEBUG:teuthology.nuke:shortname: smithi057 2024-03-29T10:52:37.561 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:52:37.589 DEBUG:teuthology.nuke:shortname: smithi179 2024-03-29T10:52:37.590 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:52:37.594 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622641', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:18:40.038645', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOIvQcBjWXDczcrEinUPtbVNMsWv8poI/HVsMsTY3lZhdJ/G2+kuaCcn8AlPIGoyBeouP2tDxUXV2DDKLqHqJMU='} 2024-03-29T10:52:37.620 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622641', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:18:40.037293', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOAEcOUl39u6xHmX9AOi+6H5hugRXuFb1wwZYRXKWzwhnLEYf6371TH0sVoDdYEUpoblQup/LyAf0MeJ87SVye4='} 2024-03-29T10:52:37.654 INFO:teuthology.orchestra.console.smithi057:Power off 2024-03-29T10:52:37.655 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T10:52:37.673 INFO:teuthology.orchestra.console.smithi179:Power off 2024-03-29T10:52:37.674 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T10:52:37.683 DEBUG:teuthology.orchestra.console.smithi057:power off output: Chassis Power Control: Down/Off 2024-03-29T10:52:37.683 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:52:37.698 DEBUG:teuthology.orchestra.console.smithi179:power off output: Chassis Power Control: Down/Off 2024-03-29T10:52:37.699 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:52:37.709 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2024-03-29T10:52:37.723 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2024-03-29T10:52:41.710 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:52:41.724 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:52:41.824 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2024-03-29T10:52:41.835 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2024-03-29T10:52:45.826 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:52:45.835 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:52:45.937 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2024-03-29T10:52:45.945 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2024-03-29T10:52:49.942 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:52:49.955 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:52:50.056 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is off 2024-03-29T10:52:50.068 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is off 2024-03-29T10:52:50.158 INFO:teuthology.orchestra.console.smithi057:Power off completed 2024-03-29T10:52:50.169 INFO:teuthology.orchestra.console.smithi179:Power off completed 2024-03-29T10:52:50.274 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1389.2150809764862 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=f9aaef0fd383456a9511deef9a1dbe1b status: fail success: false 2024-03-29T10:52:50.274 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:52:50.353 INFO:teuthology.run:FAIL